- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SENSA, a domain-aware method for improving information retention in LLM-based summarization. It first generates summaries of development documents and uses LLM-generated questions and an LLM judge to identify information lost during compression. These failures are grouped into categories, and the three most common categories are added to the summarization prompt as preservation rules. During inference, documents are split into sections and summarized separately using these rules and proportional token budgets. Experiments show that SENSA generally outperforms existing compression methods, especially on coding tasks.
Critical strengths
-
The method is simple and interpretable. The learned rules are short and human-readable, are learned once per domain, and need far fewer model calls than baselines.
-
The experimental setup is thorough and careful. All methods must use 90–100% of the total length budget, so reductive and generative methods are compared at nearly the same length. All prompt examples are also shown in full.
Critical weaknesses
1. Limited validation of the evaluation metric. The paper relies on LLM-generated questions and an LLM judge to measure information retention. Although the authors report high agreement between two judges, this does not establish whether the metric reflects human judgments or downstream task performance. More direct validation would strengthen the evaluation.
2. Unclear practical relevance of the generated questions. The questions are generated from the source documents without considering how the compressed information will be used. It is unclear whether they capture information that matters for real tasks or simply reflect the question generator's preferences. This is particularly concerning for the coding datasets, where retaining more answerable questions does not necessarily mean better agent performance.
3. Potential bias in the evaluation pipeline. The question generator and summarizer use the same model, while the primary judge belongs to the same model family. Although the authors evaluate agreement with a second judge, they do not examine whether using a different question generator would change the results or the ranking of compression methods.
4. Unjustified assumptions in section-wise compression. SENSA allocates token budgets to sections based on their lengths and applies the same preservation rules to every section. However, longer sections are not necessarily more important, and different sections may contain different types of information worth preserving. The paper does not explain why these choices are appropriate or compare them with alternative budget allocation and section-specific compression strategies. This raises concerns about how well the approach would work in real-world tasks.
Questions for authors
-
Do the authors have any evidence that the QA-based information retention metric correlates with human judgments or downstream task performance? This would help clarify whether the reported improvements translate into practical benefits.
-
How sensitive are the results to the choice of question generator? Would using a model from a different family change the generated questions or the relative performance of the compression methods?
-
What proportion of summarization failures is covered by the top three categories (shown in Fig. 3) in each domain? How does performance change as the number of preservation rules increases? These results would help assess the paper's central claim that summarization errors have a low-dimensional structure.
-
What is the practical motivation for the QA-based benchmark used in this paper? Why should performance on automatically generated questions be considered a meaningful measure of context compression quality? In particular, how does this benchmark relate to the information needs of real users or downstream agents?