- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this paper, the authors propose ForeSift, a training-free prompt compression framework designed to reduce the inference cost of large language models while preserving reasoning-relevant information. To address the limitations of conventional compression methods that select context before reasoning, ForeSift leverages a small language model to extract question-oriented and reasoning-oriented attention distributions through a single prefill pass without generating explicit reasoning traces. The authors introduce an agreement-aware fusion mechanism that adaptively combines the two attention views based on their consistency, followed by a sentence-level selection strategy to preserve contextual coherence under a strict token budget. Extensive experiments on LongBench and RULER across different model families and compression budgets demonstrate improvements over existing compression methods, achieving competitive task performance with reduced inference latency and memory consumption.
Critical strengths
The paper identifies the limitations of conventional prompt compression methods that select context before reasoning and proposes an efficient approach that leverages reasoning-oriented attention to better preserve task-relevant information without additional training or draft decoding.
Critical weaknesses
-
The methodological novelty appears limited. The core idea of using attention scores from a small model to guide token reduction for a larger model has already been explored in prior work, such as SGL [1]. Although ForeSift extends this paradigm to textual prompt compression and introduces an agreement-aware fusion mechanism, the contribution beyond existing approaches remains relatively incremental.
-
The experimental evaluation is limited to LongBench and RULER, without including challenging mathematical or code reasoning benchmarks such as MATH500, AIME, or LiveCodeBench. Since the method is motivated by preserving intermediate evidence required for reasoning, the lack of evaluation on reasoning-intensive tasks weakens the support for its central claims.
Reference
[1] A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs. CVPR, 2025.
Questions for authors
-
How are the required attention scores extracted? Since optimized attention implementations such as FlashAttention avoid materializing attention matrices, does the extraction introduce additional computational or memory overhead, and is this overhead included in the reported efficiency results?
-
Why does the proposed fusion mechanism assign greater weight to the reasoning-oriented attention when the two attention views are more consistent? In multi-hop reasoning, disagreement may instead indicate complementary evidence that is important for answering the question.
-
Since ForeSift does not explicitly generate reasoning traces, how do the authors distinguish reasoning-relevant attention from attention variations induced by different prompt formulations?
-
How do the reported efficiency improvements vary with output length and batch size, particularly under optimized inference frameworks such as vLLM or SGLang?