- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
SABR is a training-free selector that combines frozen semantic scores with local coordination, redundancy and proximity features, and exact generator-token packing. The paper compares it with a reranker at matched realized context and prompt lengths, and reports separate analyses of answer quality, evidence retention, and runtime.
Critical strengths
The paper is careful about a real confound in context-selection experiments: retaining more tokens can make a method look better. The exact-token implementation and the separation of primary, confirmation, diagnostic, and timing cohorts are clearly described. The authors also report negative or inconclusive findings instead of presenting every diagnostic as a success.
Critical weaknesses
The central claim is not supported at the level needed for acceptance. The strongest positive result is a localized SQuAD improvement on a primary reserve that was inspected during method development. The four-task macro effect is uncertain on that reserve and slightly negative on the independent cohort; the independent confirmation also uses only one generator. Thus the paper does not establish that SABR improves context selection beyond the chosen SQuAD setting.
The method contains several hand-designed decisions—eight-word fragments, mixing coefficients, route rules, neighborhood bonuses, and the 70% operating point—that were developed using public data. The independent cohort reduces but does not remove this development dependence. The component interventions change multiple parts of the policy or use a separate matched-prefix adapter, so they do not identify which claimed structural mechanism is responsible. The runtime experiments are small and show that SABR is not faster than full context in the reported settings. Taken together, the work is a careful analysis of a promising heuristic, but its general methodological conclusion is too strong for the evidence provided.
The related-work coverage is incomplete. The paper should also discuss the following directly relevant works:
- COMI: Coarse-to-Fine Context Compression via Marginal Information Gain — ICLR 2026
- GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment — ACL 2026
- In-context Autoencoder for Context Compression in a Large Language Model — arXiv, 2023
- Learning to Compress Prompts with Gist Tokens — NeurIPS 2023
- Adapting Language Models to Compress Contexts — EMNLP 2023
Without these references, the relationship between SABR’s token-selection heuristic and prior learned or latent context-compression methods is not sufficiently clear.
Questions for authors
Can you evaluate one frozen configuration on a genuinely untouched, question-disjoint test set with more than one generator? How were the fragment size, coefficients, route rules, and operating point selected relative to the inspected reserve? Please report whether the same conclusions hold under paragraph- or sentence-level segmentation and under a fully matched end-to-end cost budget.
Thank you for pointing out these issues. Your questions helped us distinguish choices made during development from evidence on held-out questions.
For your first question about untouched questions and multiple generators, we completed a frozen evaluation. We fixed one configuration before running Qwen2.5-7B and Mistral-7B on 512 SQuAD questions from 177 articles. The questions come from the public training split, with the official validation split and training articles in recovered development/evaluation records excluded. We found zero overlap in question IDs, normalized questions and paragraph hashes with those records. Checkpoint pretraining exposure is unknown. At the 70% cap all arms matched realized context and prompt tokens. SABR minus reranker answer F1 was +3.15 [0.46, 5.88] for Qwen and +1.62 [−0.70, 3.99] for Mistral. Article-cluster intervals were [0.37, 5.96] and [−0.90, 4.10]. The Qwen gain survives both analyses. The primary four-task macro was +0.92 [−0.08, 1.91] and the independent 896-question macro was −0.34 [−2.40, 1.76]. Neither supports an overall QA gain.
For your second question about design choices, we now state the development history explicitly. Eight-word fragments, routing rules and bonuses were hand-designed. The inspected reserve informed the mixing coefficients. We tested 50% and 70% and highlighted 70% after seeing positive SQuAD estimates for all three generators. In the frozen 512-question test, disabling only the four adaptive terms changed F1 little (SABR minus control +0.20/−0.09, Qwen/Mistral). A stronger control held semantic mixing and packing fixed and disabled grouping, smoothing, bridging, proximity, redundancy and the boundary prior together. On its 709-question sample, full SABR minus this control was +2.86 [0.56, 5.11] for Qwen and +1.24 [−0.75, 3.28] for Mistral. This supports a joint contribution for Qwen on these questions.
For your third question about segmentation and end-to-end cost, we tested sentence selection and matched end-to-end time separately. SABR minus reranker was +3.40/+1.19 on the coordination sample and +1.55/+0.53 on the 19-arm component sample. Each has 709 questions, sharing all 28 articles and 186 question IDs, and share no article with the 512-question test. We report them separately. On the component sample, sentence selection minus the fragment policy was +0.23 [−1.73, 2.26] and +1.02 [−0.84, 2.84]. Neither interval resolves a sentence advantage. We plan to compare paragraph and fragment selection on HotpotQA at equal token budgets. No alternative mixing weight, radius or density threshold cleared its within-family Holm correction. Appendix I includes the complete contrasts.
We plan to compare paragraph and fragment selection on HotpotQA at equal token budgets and will post the results of that comparison here.
Matched end-to-end time. We calibrated the reranker’s cap using time alone on 96 separate questions, then locked 100% for Qwen and 80% for Mistral against SABR at 70%. On all 709 component-study questions, mean reranker/SABR request-time ratios were 1.016 and 0.996, both within the prespecified 5% tolerance. Timing included online helper scoring, packing and natural answer generation. SABR minus reranker was −4.37 F1 for Qwen and −0.81 for Mistral. The two-model mean was −2.59 [−4.49, −0.81] by paired article bootstrap. Thus the token-matched advantage does not carry over to this time budget.
Thank you also for the related work you suggested. Section 2.1 discusses all five suggested works. COMI [1] directly shares the relevance–redundancy motivation. GMSA [2] merges tokens into soft representations. Compact representations are also learned by ICAE [3], Gist Tokens [4] and AutoCompressors [5]. SABR retains source text for a frozen generator. These methods were not benchmarked head-to-head here.
[1] Tang et al. COMI: Coarse-to-Fine Context Compression via Marginal Information Gain. ICLR 2026. [2] Tang et al. GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment. ACL 2026. [3] Ge et al. In-context Autoencoder for Context Compression in a Large Language Model. ICLR 2024. [4] Mu et al. Learning to Compress Prompts with Gist Tokens. NeurIPS 2023. [5] Chevalier et al. Adapting Language Models to Compress Contexts. EMNLP 2023.
We welcome questions during the discussion. We hope these revisions and clarifications will be helpful in your later assessment of the paper.