acceptodds
Withdrawn from ICLR 2027

ForeSift: Speculative Reasoning for Small-to-Large Prompt Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long prompts raise target prefill and KV-cache costs, motivating evidence-preserving compression without modifying the target. Small–large compressors let a lightweight draft inspect the source before one target call, but their static context-, question-, or lookahead-conditioned views can miss evidence needs that become clearer as reasoning unfolds—a selection-before-reasoning mismatch. ForeSift addresses this mismatch in one causal draft prefill: it keeps the request-conditioned profile as a Task Anchor, scales a latent reasoning view by its window-level agreement with the anchor, and selects complete sentences under the exact target-tokenizer budget. The method is training-free, requires no draft decoding, and invokes the target once. Across LongBench and RULER- 32K, two model families, and three budgets, ForeSift achieves the best compressed-method macro in all 12 settings, with especially large gains at the 512-token budget and on variable tracing and aggregation. At 32K, a pure-Transformers deployment provides a 1.71× end-to-end speedup and reduces additional allocation beyond the shared target residency by 73% relative to Dense inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt compression, Long-context language models, Speculative reasoning, Small-to-large inference, Efficient LLM inference

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_rF7ROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors propose ForeSift, a training-free prompt compression framework designed to reduce the inference cost of large language models while preserving reasoning-relevant information. To address the limitations of conventional compression methods that select context before reasoning, ForeSift leverages a small language model to extract question-oriented and reasoning-oriented attention distributions through a single prefill pass without generating explicit reasoning traces. The authors introduce an agreement-aware fusion mechanism that adaptively combines the two attention views based on their consistency, followed by a sentence-level selection strategy to preserve contextual coherence under a strict token budget. Extensive experiments on LongBench and RULER across different model families and compression budgets demonstrate improvements over existing compression methods, achieving competitive task performance with reduced inference latency and memory consumption.

Critical strengths

The paper identifies the limitations of conventional prompt compression methods that select context before reasoning and proposes an efficient approach that leverages reasoning-oriented attention to better preserve task-relevant information without additional training or draft decoding.

Critical weaknesses
  1. The methodological novelty appears limited. The core idea of using attention scores from a small model to guide token reduction for a larger model has already been explored in prior work, such as SGL [1]. Although ForeSift extends this paradigm to textual prompt compression and introduces an agreement-aware fusion mechanism, the contribution beyond existing approaches remains relatively incremental.

  2. The experimental evaluation is limited to LongBench and RULER, without including challenging mathematical or code reasoning benchmarks such as MATH500, AIME, or LiveCodeBench. Since the method is motivated by preserving intermediate evidence required for reasoning, the lack of evaluation on reasoning-intensive tasks weakens the support for its central claims.

Reference

[1] A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs. CVPR, 2025.

Questions for authors
  1. How are the required attention scores extracted? Since optimized attention implementations such as FlashAttention avoid materializing attention matrices, does the extraction introduce additional computational or memory overhead, and is this overhead included in the reported efficiency results?

  2. Why does the proposed fusion mechanism assign greater weight to the reasoning-oriented attention when the two attention views are more consistent? In multi-hop reasoning, disagreement may instead indicate complementary evidence that is important for answering the question.

  3. Since ForeSift does not explicitly generate reasoning traces, how do the authors distinguish reasoning-relevant attention from attention variations induced by different prompt formulations?

  4. How do the reported efficiency improvements vary with output length and batch size, particularly under optimized inference frameworks such as vLLM or SGLang?

Reviewer_gVfeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposed a prompt compression method to accelerate inference and reduce memory usage.

Critical strengths

None

Critical weaknesses
  1. The presentation of the entire paper is poor. The layout of the paper, as well as the format and arrangement of the charts, does not meet the acceptance standards.
  2. Appendix A, which was repeatedly mentioned at the end of the paper, was not attached to the end of the paper, and the link was also invalid.
  3. The related work is not comprehensive. Most of the works discussed are centered around research conducted in 2023-2024, which is insufficient for the current situation. Moreover, I did not find any reference section in the submitted main manuscripts.
  4. Overall, based on the quality of the paper, I also have doubts about the soundness of the proposed methods and experiments.
Questions for authors

See Critical Weaknesses.

In my view, the paper's overall quality does not meet the submission standards.

Reviewer_nWhnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

ForeSift is a training-free prompt compression method that combines question-conditioned and fixed-hint-conditioned attention from one small-model prefill. A window-level agreement coefficient controls the hint contribution, and complete sentences are selected under a target-tokenizer budget. The paper evaluates two model pairs on LongBench and RULER and reports improvements over SpecPC and LLMLingua-2.

Critical strengths

The deployment interface is practical: compression requires no training or draft decoding and produces a conventional prompt for one unmodified target call. The reported improvements span both evaluated model pairs and several budgets, including substantial gains on variable tracing. The paper also explicitly acknowledges several limitations of its comparisons.

Critical weaknesses

The claimed contribution is not isolated. The fixed hint does not establish a reasoning-specific benefit, and suitable matched controls are missing. Comparisons with SpecPC also change selection granularity, leaving the source of improvement unclear. Adaptive confirmation is insufficiently justified. Its agreement coefficient can remain high for disjoint, diffuse distributions. In Table 4, the confidence interval for its improvement over Question-only includes zero, leaving its advantage over this simpler variant inconclusive.

Questions for authors

With identical sentence units, rendering, and budget enforcement, how do SpecPC scoring, Question-only scoring, and ForeSift compare?

Does H1 outperform a neutral sentence-selection instruction matched for question repetition, length, position, and placeholders?

What is the observed distribution of (c), and does it predict the per-example benefit of adding the hint? How does it compare with a distribution-overlap measure?

Can the confirmation ablation be repeated on RULER variable tracing and aggregation, across both model pairs?

How do latency and GPU-seconds compare under matched hardware resources, including SpecPC? Appendix A specifies three GPUs for Dense and an additional fourth GPU for ForeSift.

What causes the missing outputs underlying the UID intersections, and do the conclusions hold on a fixed evaluation set with failures explicitly accounted for?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.