acceptodds
Under review as a conference paper at ICLR 2027

BridgeComp: Prompt Compression via Reasoning-Conditioned Attention and Anchor-Conditioned Policy Optimization

Abstract

The proliferation of large language models in long-context applications has made efficient inference paramount, and prompt compression has emerged as a prominent paradigm to condense context while preserving task-relevant information. However, contemporary compression frameworks introduce semantic distortion and hallucinations by underestimating reasoning-critical tokens or inflating token importance through superficial word-matching. Moreover, answer-level supervision alone provides only indirect guidance for preserving reasoning-critical evidence. To address these challenges, we introduce BridgeComp, a prompt compression framework that estimates token importance from latent tokens that serve as implicit reasoning states. We append learnable latent tokens to the question and use upper-layer attention to isolate deep relational reasoning signals from shallow lexical noise. To align token selection with both answer prediction and compression quality, we train the compressor with reinforcement learning that jointly rewards answer correctness and supporting-evidence preservation. Experiments on question-answering and long-context benchmarks show that BridgeComp achieves state-of-the-art performance. Under a compression setting retaining only 10% of tokens, BridgeComp improves Exact Match by up to 62.44% over the strongest baseline. For long-context compression, BridgeComp achieves up to an 11.75 compression speedup and reduces peak memory usage by up to 39.14%. Our code is available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt compression, large language models, reasoning-conditioned attention, reinforcement learning, multi-hop question answering, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RyK9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents BridgeComp, a prompt compression framework that uses learnable latent tokens and upper-layer attention to pick out the information that really matters for reasoning, while cutting down on lexical-overlap bias. It also adds anchor-conditioned policy optimization to improve token selection, using answer-quality and evidence-preservation rewards. Experiments on multi-hop QA and long-context benchmarks show better downstream accuracy, evidence preservation, and compression efficiency than existing baselines.

Critical strengths
  • Comprehensive experiments and supporting evidence. The paper runs detailed experiments across multiple benchmarks and backs them up with extensive ablations and analyses. The empirical evidence is fairly convincing and supports the effectiveness of the proposed method.

  • Clear writing and good presentation. The paper is well written and easy to follow, with a clear structure and detailed explanations. The figures are also nicely designed—especially Figures 1 and 2, which do a good job of illustrating the motivation and the overall framework.

  • Well-defined problem and motivation. The paper points out meaningful limitations of existing prompt compression methods, especially the loss of reasoning-critical information. The problem is clearly formulated, and the proposed approach is well motivated.

Critical weaknesses
  • Limited methodological novelty. The problem is well defined and the motivation is convincing, but the proposed method mainly brings together existing techniques, namely learnable latent tokens, upper-layer attention aggregation, and RL-based optimization. The overall framework comes across as somewhat of a composite of multiple techniques, so the methodological novelty is relatively limited.

  • Concerns about baseline comparisons and evaluation fairness. It is unclear whether the baselines receive supervision comparable to BridgeComp, which uses additional supervision from gold supporting documents during RL training. Moreover, the efficiency comparisons mostly focus on compression latency rather than end-to-end inference cost. These factors make it difficult to fully assess the performance and efficiency advantages of the proposed method.

Questions for authors
  • What evidence supports the claim that the learned latent tokens capture reasoning-critical information, rather than simply learning better task-specific relevance representations?

  • BridgeComp uses gold supporting-document annotations in its RL rewards. Do the competing baselines have access to comparable supervision, and how much of the performance improvement can be attributed to this additional supervision?

5 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer RyK9

Thank you for your constructive feedback and for recognizing the motivation and empirical evaluation of our work. We clarify BridgeComp’s methodological contribution and present additional controlled experiments addressing supervision fairness, bridging-evidence preservation, and end-to-end efficiency. We will incorporate these clarifications and results into the revised manuscript.


W1. Methodological novelty of BridgeComp

To the best of our knowledge, BridgeComp is the first prompt-compression framework that jointly integrates learnable latent tokens, upper-layer attention aggregation, and anchor-conditioned policy optimization. While individual components have been explored in other domains, adapting and unifying them effectively for budget-constrained prompt compression is nontrivial. Rather than loosely stacking independent modules, BridgeComp couples learned token scoring with policy optimization through a shared latent-to-document attention distribution, which simultaneously defines token importance, the compression anchor, and the exploration policy under a fixed token budget. Our design resolves two mismatches that arise when combining these paradigms naively.

  1. Bridging latent tokens to discrete token selection (RCA & LSA): Prior latent token methods typically feed continuous soft embeddings into a coupled generator, restricting downstream compatibility. Meanwhile, existing attention-based prompt compression methods have never used latent tokens. In contrast, RCA and LSA allow learned latent representations to guide multi-hop evidence selection while passing discrete compressed text to any black-box downstream generator.
  2. Budget-based exploration aligned with inference (AGPG): Standard RL methods sample unconstrained masks from scratch, suffering from high variance and token-budget drift. Although prior prompt compression methods such as TACO-RL have employed RL for token selection, AGPG differs by aligning budget-constrained exploration with deterministic inference-time selection. Specifically, AGPG constructs the exact deterministic top-k mask used at inference as an anchor and samples local alternatives by exchanging equal numbers of retained and discarded tokens. Every candidate mask strictly satisfies the target budget, ensuring that reward gains cannot arise from retaining extra tokens, while computing advantages relative to both the sampled group and the anchor directly aligns model training with inference-time selection.

The methodological contribution therefore lies not in the individual components, but in their integration through a shared attention signal that connects learned evidence scoring, budget-constrained token selection, and anchor-relative policy optimization. In the revised manuscript, we will sharpen our presentation in Sections 1 and 3 to make this unified coupling, and its distinction from prior soft-token and unconstrained RL approaches more explicit.

AuthorsOpenReview ↗

Response 2

W2 & Q2. Baseline supervision, its contribution, and end-to-end efficiency

Training configurationSupporting-document rewardsHotpotQA EM/F12Wiki EM/F1MuSiQue EM/F1
Full rewardrattn,pmissr_{attn},p_{miss}66.4/80.064.3/72.846.3/60.2
w/o rattnr_{attn}pmissp_{miss}63.2/77.361.8/69.144.1/58.0
w/o pmissp_{miss}rattnr_{attn}65.3/79.363.8/71.746.0/59.6
Answer-only RLnone63.0/77.462.1/70.244.0/57.5

Table R1. QA performance across reward configurations at τ=0.5\tau=0.5. Answer-only RL removes both supporting-document rewards while keeping NTP, AGPG, λRL\lambda_{RL}, and the remaining training settings fixed.

MethodCompression RateCompression (s)Generation (s)E2E Latency (s)
Original Prompt--6.646.64
AttnComp0.517.322.2219.55
CPC0.512.892.1215.02
BridgeComp0.52.122.124.24
AttnComp0.214.471.2115.68
CPC0.211.161.2412.39
BridgeComp0.21.851.213.05

Table R2. Compression, generation, and end-to-end latency with 20K document tokens before compression. Generation includes generator prefill and decoding.

Several baselines use additional supervision from gold supporting documents. BRIEF-Pro and CompAct incorporate such supervision during training, though their supervision signals differ from those utilized by BridgeComp. BRIEF-Pro constructs target summaries using gold supporting documents, while CompAct is trained on teacher-generated compression outputs, and completeness judgments from retrieved contexts and distractor contexts. In contrast, BridgeComp incorporates supporting-document annotations into its RL rewards.

Furthermore, we measure how much of the performance improvement can be attributed to this additional supervision. Table R1 quantifies the contribution of the two supporting-document rewards by comparing BridgeComp (i.e., Full reward) and its variant Answer-only RL. The variant “Answer-only RL” removes both rewards while keeping NTP, AGPG, λRL\lambda_\text{RL}, and the remaining training settings unchanged. Adding two rewards improves EM by 3.4, 2.2, and 2.3 points and F1 by 2.6, 2.6, and 2.7 points on HotpotQA, 2WikiMultihopQA, and MuSiQue, respectively. “Answer-only RL” still outperforms AttnComp on all three datasets, where AttnComp results are reported in Table 1 of the manuscript. On MuSiQue, this variant achieves 44.0/57.5 EM/F1, compared with 38.7/54.4 for AttnComp, retaining gains of 5.3 EM and 3.1 F1 points. Thus, the supporting-document rewards provide additional benefits, while the advantage over AttnComp persists when both rewards are removed.

To fully assess the performance and efficiency advantages of the proposed method, we measure the end-to-end inference cost. Table R2 reports end-to-end latency, which consists of latencies of compression and the complete generator call from prompt submission until the response is finished. We use original retrieved documents with up to 20K tokens before compression to more clearly examine the trade-off between compression overhead and downstream generation latency. BridgeComp reduces total latency from 6.64 seconds for the uncompressed prompt to 4.24 seconds at τ = 0.5 and 3.05 seconds at τ = 0.2, corresponding to reductions of 36.1% and 54.1%, respectively. The downstream generation-time savings therefore exceed the compression cost in this setting. We will incorporate these measurements into Appendix E.

AuthorsOpenReview ↗

Response 3

Q1. Evidence that latent attention preserves reasoning-critical information

Backbone LLMAttention sourceBridging Entity Recall (%)Evidence-triple Retention (%)
LLaMA3.2-3BQ→DQ\rightarrow D92.8283.2
LLaMA3.2-3BR→DR\rightarrow D98.7790.8
Qwen2.5-7BQ→DQ\rightarrow D79.6670.6
Qwen2.5-7BR→DR\rightarrow D96.2491.6

Table R3. Bridging Entity Recall and evidence-triple retention on 2WikiMultihopQA at τ=0.5\tau=0.5. Within each backbone, the checkpoint, input documents, selected layers, and compression budget are fixed; only the attention source changes.

To distinguish multi-hop bridging preservation from task-specific relevance, we conduct a controlled comparison between question-to-document (i.e., Q→DQ\rightarrow D) and latent-to-document (i.e., R→DR\rightarrow D) attention scores on 2WikiMultihopQA at τ = 0.5. Both scoring sources are extracted from the same trained checkpoint under matched compression conditions. Using gold evidence triples provided in 2WikiMultihopQA, we identify intermediate entities in these triples that appear in the input documents but are absent from the query, and evaluate two micro-averaged metrics across the entire test set: (1) Bridging Entity Recall measures the proportion of ground-truth intermediate entities retained after compression; (2) Evidence-triple retention measures the proportion of gold evidence triples that remain semantically supported in the compressed context, evaluated by GPT-6.1-Terra.

As shown in Table R3 above, R→DR\rightarrow D preserves more annotated intermediate entities and evidence relations than Q→D across both backbones. For LLaMA3.2-3B, R→DR\rightarrow D improves Bridging Entity Recall by 6.41% (92.82% → 98.77%) and evidence-triple retention by 9.13% (83.2% → 90.8%). For Qwen2.5-7B, the corresponding improvements are 20.81% (79.66% → 96.24%) and 29.75% (70.6% → 91.6%). Because bridging entities have no direct lexical overlap with the question, these gains demonstrate that latent-token attention effectively preserves intermediate entities and relational evidence required for multi-hop inference. In the revised manuscript, we will include this empirical analysis and refine our phrasing to explicitly frame this effect as preserving annotated bridging entities and evidence relations, avoiding any claim that latent tokens explicitly encode identifiable intermediate reasoning steps.

Reviewer_wa4qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BridgeComp compresses prompts by scoring document tokens using attention from learnable latent tokens, aggregated over selected upper transformer layers. The compressor is trained with next-token prediction and anchor-conditioned group policy gradient (AGPG), using rewards for answer quality, attention retained over gold supporting documents, and supporting-document coverage. The paper reports consistent improvements on multi-hop QA and long-context benchmarks.

Critical strengths
  1. The empirical improvements are consistent across datasets, compression budgets, and downstream generators. At a retention ratio of τ=0.2\tau=0.2, BridgeComp outperforms AttnComp on all three multi-hop QA datasets with both generators (Table 1). With GPT-5.6 Luna, the EM scores increase from 58.8 to 64.9 on HotpotQA, 55.6 to 58.3 on 2WikiMultihopQA, and 33.8 to 41.9 on MuSiQue.

  2. The paper provides substantial ablations and auxiliary analyses. Table 5 and Appendix D.5 support the usefulness of RCA, LSA, and the training objectives and reward components. Appendices F.1 and F.6 also assess evidence preservation beyond answer accuracy, strengthening the empirical case for the framework.

Critical weaknesses
  • Table 5 and Appendix G.2 show that AGPG outperforms GPG, but Eq. (9) changes both the baseline and the advantage scale: Ai=(1+η)(ri−rˉ)+η(rˉ−ra),A_i=(1+\eta)(r_i-\bar r)+\eta(\bar r-r_a), where rar_a is the anchor reward. The comparison therefore leaves open whether the gain comes from the anchor baseline or the changed RL update scale. Appendix G.12 varies the RL-loss weight only for AGPG. Please compare both methods after separately tuning their RL-loss weights. Loss normalization should also be matched: Appendix C.1 averages GRPO over 2b2b selections, whereas AGPG sums their log-probabilities.

  • Section 5.7 and Appendices F.4 and F.8 show that latent tokens help identify supporting evidence. They do not establish whether these tokens encode intermediate reasoning steps or simply aggregate task-relevant information more effectively. The paper should provide a direct analysis of intermediate entities or relations, or describe the latent tokens more cautiously. Appendices F.1 and F.6 already support improved reasoning-chain preservation, but they evaluate only the complete framework. The reward ablations in Appendix D.5 report QA scores without showing which evidence is retained. This matters because the auxiliary rewards use document-level support labels: retaining one token avoids a document's miss penalty, even if its necessary facts are removed. Measuring supporting-fact or evidence-triple retention for these ablations would clarify what the rewards improve.

  • Appendix E measures compression latency. Please also report total latency, including generator prefill and decoding, and include an uncompressed-prompt baseline. This would show whether the downstream time savings outweigh the added compression cost.

Questions for authors

Please see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer wa4q

Thank you for your thoughtful and constructive feedback and for recognizing our contributions. We appreciate the time and effort you devoted to reviewing our work. Below, we clarify the RCA mechanism and present additional experiments on bridging information preservation. We will incorporate these clarifications and findings into the revised manuscript. We hope our responses clarify the connection between our motivation and the proposed method, thus addressing your concerns. We welcome further questions or suggestions.


W1. Independent tuning of RL-loss weights and loss normalization

BackboneLossλRL\lambda_{\mathrm{RL}}HotpotQA EMHotpotQA F12Wiki EM2Wiki F1MuSiQue EMMuSiQue F1
LLaMA3.2-3BNTP + GRPO0.164.077.360.270.043.055.9
NTP + GPG0.464.277.861.870.943.355.5
NTP + GPG (Scale-only)0.464.377.562.071.143.555.8
NTP + AGPG0.566.480.064.372.846.360.2
Gemma2-2BNTP + GRPO0.164.578.958.967.642.356.9
NTP + GPG0.464.879.058.567.242.457.1
NTP + GPG (Scale-only)0.465.079.458.867.242.557.3
NTP + AGPG0.565.179.559.768.443.257.7

Table R1. QA performance of NTP combined with GRPO, GPG, GPG (Scale-only), or AGPG using independently selected RL-loss weights across two compressor backbones.

To disentangle advantage scaling from the anchor-dependent baseline and ensure a fair comparison across loss conventions, we conduct a controlled comparison of GPG, GRPO, AGPG, and a new Scale-only GPG variant after independently sweeping λRL\lambda_{\mathrm{RL}} for each objective.

  • Independent λRL\lambda_{\mathrm{RL}} sweep: For every method, we independently sweep λRL∈0.01,0.05,0.1,0.2,…,0.9\lambda_{\mathrm{RL}} \in \\{0.01, 0.05, 0.1, 0.2, \ldots, 0.9\\}. To give the baselines the strongest possible comparison, we report each baseline's peak performance across the sweep, whereas for AGPG we select λRL=0.5\lambda_{\mathrm{RL}} = 0.5 as a representative setting within its stable performance plateau. As shown in Table R1, AGPG achieves the highest EM and F1 across all three datasets and both compressor backbones (LLaMA3.2-3B and Gemma2-2B), with consistent trends across downstream generators (Qwen2.5-7B and Qwen3.5-4B). On MuSiQue with LLaMA3.2-3B, AGPG outperforms tuned GPG by 3.0 EM and 4.7 F1 points.

  • Isolating advantage scaling from the anchor term: The Scale-only variant (NTP + GPG (Scale-only)) applies the multiplicative scaling of AGPG without the anchor baseline: Aiscale=(1+η)(ri−rˉ)A_i^{\mathrm{scale}} = (1+\eta)(r_i-\bar{r}). For both compressor backbones, this variant performs nearly identically to standard GPG, within 0.3 EM and 0.4 F1 points, whereas AGPG substantially outperforms the variant across all datasets (e.g., 46.3 EM / 60.2 F1 vs. 43.5 EM / 55.8 F1 on MuSiQue with LLaMA3.2-3B). These results indicate that AGPG's gains cannot be explained solely by advantage rescaling and support the additional contribution of the anchor-relative term η(rˉ−ra)\eta(\bar{r}-r_a).

  • Matching loss normalization: Because every rollout contains a fixed number of 2b2b selection decisions, the summed log-probability objective of AGPG is an exact constant multiple (2b2b) of its per-selection mean. Expressed on a unified per-selection mean basis, sweeping λRL∈[0.01,0.9]\lambda_{\mathrm{RL}} \in [0.01, 0.9] corresponds to loss weights of [0.01,0.9][0.01, 0.9] for GRPO and [0.01⋅2b,0.9⋅2b][0.01 \cdot 2b, 0.9 \cdot 2b] for AGPG. Importantly, both GPG and GRPO reach their peak performance within the sweep range (at λRL≤0.4\lambda_{\mathrm{RL}} \leq 0.4, with performance degrading toward 0.9), indicating that their performance is not bottlenecked by the upper bound of the sweep.

In the revised manuscript, we will include the Scale-only GPG variant, report the full λRL\lambda_{\mathrm{RL}} sweep curves for all methods, and align all objectives under an explicit per-selection mean normalization convention.

AuthorsOpenReview ↗

Response 2

W2. Evidence preservation across reward ablations and interpretation of latent tokens

ModelQ→D Recall (%)R→D Recall (%)Improvement (%p)95% Bootstrap CI (%p)
LLaMA3.2-3B92.8298.775.95[4.43,7.47]
Qwen2.5-7B79.6696.2416.58[14.32,19.06]

Table R2. Bridging Entity Recall (%) on 2WikiMultihopQA at τ=0.5\tau=0.5. Comparison of question-to-document (Q→DQ\rightarrow D) and latent-to-document (R→DR\rightarrow D) attention across two LLM backbones. Improvement denotes the absolute recall gain of R→DR\rightarrow D over Q→DQ\rightarrow D, with 95% confidence intervals estimated using paired bootstrap resampling.

Reward ConfigurationEMF1Supporting-document coverage (%) ↑Evidence-triple retention (%) ↑
Full reward64.372.899.890.8
w/o rF1r_{F_1}62.970.297.986.3
w/o rattnr_{attn}61.869.196.182.3
w/o pmissp_{miss}63.871.796.885.8

Table R3. QA performance, supporting-document coverage, and evidence-triple retention across reward ablations on the same 2WikiMultihopQA test set used in the main results, at τ=0.5\tau=0.5.

To address both points, we conduct new analyses on 2WikiMultihopQA (τ=0.5\tau=0.5) examining intermediate entity preservation across attention sources, and factual triple retention across reward ablations below:

1. Intermediate entity preservation: As shown in Table R2, we compare question-to-document (Q→DQ\rightarrow D) and latent-to-document (R→DR\rightarrow D) attention while holding the trained checkpoint, input documents, selected upper layers, and compression budget fixed. Using gold evidence triples, we identify intermediate entities that connect evidence relations, appear in the original documents, and are absent from the question. Bridging Entity Recall measures the proportion of these entities retained after compression, micro-averaged across all eligible bridging entities. R→DR\rightarrow D consistently outperforms Q→DQ\rightarrow D in Bridging Entity Recall, improving recall by 6.41% (92.82% →\rightarrow 98.77%) on LLaMA3.2-3B and 20.81% (79.66% →\rightarrow 96.24%) on Qwen2.5-7B. These results indicate that latent-token attention more effectively retains intermediate entities that question-based attention may overlook. We agree that improved evidence retention does not by itself establish that latent tokens internally encode explicit multi-step reasoning traces. We will include Table R2 in Section 5.7, and revise our claims throughout the paper to emphasize their role in preserving intermediate entities and relational evidence.

2. Evidence triple retention across reward ablations: Table R3 evaluates all four reward configurations in Appendix D.5 using two micro-averaged metrics over the full test set:

  • Supporting-document coverage measures the fraction of gold supporting documents present in the original input that retain at least one token after compression.
  • Evidence-triple retention measures the fraction of gold triples supported by the original input that remain semantically supported after compression. Gold triples are obtained from the triple annotations in 2WikiMultihopQA. We use GPT-6.1-Terra to independently assess triple preservation before and after compression, applying the same evaluation protocol across all four reward configurations.

Retaining a single token can satisfy document coverage, whereas triple retention requires the corresponding fact or relation to remain semantically supported.

Removing any of the three components reduces evidence-triple retention. For example, compared with removing rattnr_{attn}, the full reward improves evidence-triple retention by 10.3% (82.3% →\rightarrow 90.8%). Compared with removing pmissp_{miss}, the full reward improves supporting-document coverage by 3.1% (96.8% →\rightarrow 99.8%) and evidence-triple retention by 5.8% (85.8% →\rightarrow 90.8%).

These results support improved preservation of reasoning-relevant evidence, but do not directly establish that latent tokens internally encode intermediate reasoning steps. We will revise the corresponding claims to emphasize their contribution to preserving intermediate entities and relations. We will add Table R3 and this discussion to Appendix D.5 in the revised manuscript.

AuthorsOpenReview ↗

Response 3

W3. End-to-end latency compared with uncompressed prompts

MethodCompression RateCompression (s)Generation (s)E2E Latency (s)
Original Prompt--6.646.64
AttnComp0.517.322.2219.55
CPC0.512.892.1215.02
BridgeComp0.52.122.124.24
AttnComp0.214.471.2115.68
CPC0.211.161.2412.39
BridgeComp0.21.851.213.05

Table R4. Compression, generation, and end-to-end latency with 20K document tokens before compression. Generation includes generator prefill and decoding.

Following your suggestion, we have expanded our latency analysis to measure compression latency, generation latency (covering both prefill and decoding), and end-to-end latency, including an uncompressed-prompt baseline. Generation latency is measured from prompt submission until the complete response is produced. We use 20K document tokens before compression to more clearly examine the trade-off between compression overhead and downstream generation time. As reported in Table R4 above, BridgeComp consistently reduces end-to-end latency even after accounting for compression overhead. At τ = 0.5, BridgeComp achieves a 36.1% reduction (6.64 s →\rightarrow 4.24 s) in end-to-end latency and a 1.57× speedup. At τ = 0.2, it achieves a 54.1% reduction (6.64 s →\rightarrow 3.05 s) and a 2.18× speedup. These results confirm that the downstream generation-time savings outweigh the added compression overhead in the evaluated 20K-token setting. We will add these results to Appendix E in the revised manuscript.

Reviewer_vB7sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a soft token based prompt compression algorithm to prevent static importance bias and lexical-overlap bias, as well as bridge the gap between token importance and reasoning-critical information.

Critical strengths

The paper shows the importance of preserving the defined bridging information that may appear unimportant initially but become essential during reasoning. Its combined approach effectively reduces prompt length and improves evidence preservation, achieving strong results against other baselines. The component ablations and evidence preservation analyses provide useful support for the method’s effectiveness.

Critical weaknesses

The paper provides an intuitive explanation and empirical evidence that RCA preserves more bridging information. However, the explanation of why latent token attention is particularly designed to filtering such information could be more explicit. A more detailed account of how this mechanism addresses static importance bias is necessary, which would strengthen the connection between the motivating problem and the proposed method. The clarification of this mechanism could be in favor of more generalizable method to other kinds of task.

Questions for authors

Could the authors lift on why latent token attention helps retain bridging information that question based attention may overlook? Please clarify how this relates to the static importance bias discussed in the introduction as the very first problem.

3 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer vB7s

Thank you very much for your thoughtful and constructive feedback. We sincerely appreciate your recognition of our contributions and the time and effort you invested in reviewing our work. We have carefully considered your comments and provide further clarification of the RCA mechanism, together with additional experimental results on bridging information preservation. We will incorporate these clarifications and new findings into the revised manuscript. We hope that our responses clarify the connection between our motivating problem and the proposed method, thus addressing your concerns. We welcome any further questions or suggestions.


W1. Mechanism of RCA and its connection to static importance bias

RCA is designed to mitigate static importance bias by learning an attention source aligned with answer prediction and supporting-evidence preservation. We append learnable latent tokens after the retrieved documents and a question, allowing them to be contextualized over the entire preceding hard context. NTP encourages the representations of the latent tokens useful for answer formation, while evidence-aware RL optimizes the resulting token selection through the task reward, attention quality reward, and document coverage penalty. Thus, the mechanism is not simply adding latent tokens, but training their attention-based selection to preserve information needed for answering.

Our hypothesis does not assume that question-based attention cannot capture contextual dependencies. Rather, immediate question-side salience may not reflect the importance of intermediate evidence for subsequent answer formation. RCA addresses this mismatch using learned, context-dependent latent states, allowing entities absent from the question to receive higher importance when they connect relevant evidence. This does not require explicitly updating importance at each reasoning step: compression uses a single forward pass. The LongBench results further support applicability beyond multi-hop QA, including summarization and code completion (Table 4 in Section 5.5). We will clarify this mechanism and its connection to static importance bias in the revised manuscript.


Q1. Why does latent-token attention retain bridging information that question-based attention may overlook?

ModelQ→D Recall (%)R→D Recall (%)Improvement (%p)95% Bootstrap CI (%p)
LLaMA3.2-3B92.8298.77+5.95[4.43, 7.47]
Qwen2.5-7B79.6696.24+16.58[14.32, 19.06]

Table R1: Bridging Entity Recall (%) on 2WikiMultihopQA at τ=0.5\tau = 0.5. Comparison of question-to-document (Q→DQ \rightarrow D) and latent-to-document (R→DR \rightarrow D) attention across two LLM backbones. Improvement denotes the absolute recall gain of R→DR \rightarrow D over Q→DQ \rightarrow D, with 95% confidence intervals estimated using paired bootstrap resampling.

For two backbone LLMs LLaMA3.2-3B and Qwen2.5-7B, we compare question-to-document (Q→DQ \rightarrow D) and latent-to-document (R→DR \rightarrow D) attentions to evaluate their ability to preserve bridging information on 2WikiMultihopQA at τ = 0.5.

For each backbone, we fix the trained checkpoint, input documents, selected upper layers, and compression budget, varying only the scoring source. Using gold evidence triples of the dataset, we identify intermediate entities that present in the input documents but are absent from the question. Bridging Entity Recall measures the proportion of these entities retained after compression relative to all such entities present in the original context, averaged across all evaluation samples.

As shown in Table R1, R→DR \rightarrow D consistently outperforms Q→DQ \rightarrow D in Bridging Entity Recall, yielding gains of 5.95 and 16.58 percentage points on LLaMA3.2-3B and Qwen2.5-7B, respectively. The 95% confidence intervals for these improvements, estimated using paired bootstrap resampling, range from 4.43 to 7.47 percentage points for LLaMA3.2-3B and from 14.32 to 19.06 percentage points for Qwen2.5-7B. These intervals lie entirely above zero, providing statistical support for the observed improvements. These results indicate that the learned latent attention source more effectively retains annotated bridging entities absent from the question, supporting its role in mitigating static importance bias. This comparison demonstrates stronger entity retention within the trained compressors; it does not by itself establish preservation of complete reasoning chains. We observe the same trends on HotpotQA and MuSiQue. We will include the complete results across all three benchmarks in the final manuscript.

Thanks for the reply. I will keep my positive rating.

AuthorsOpenReview ↗

Dear Reviewer vB7s

We would like to express our sincere gratitude for your careful evaluation of our work and for acknowledging its contributions. Your insightful suggestions and constructive comments have been invaluable in helping us strengthen our research. We have made every effort to thoroughly address all of your concerns and suggestions. We also plan to integrate your feedback and the supplementary experimental findings into the final manuscript to ensure a more comprehensive and rigorous presentation of our work.

We truly appreciate your time, effort, and thoughtful feedback throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.