acceptodds
Withdrawn from ICLR 2027

SENSA: Unsupervised Rule Generation for More Robust Textual Summarization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM summarization/compaction has become an increasingly important use case of LLMs as context sizes increase along with token prices. We introduce SENSA (Summarize, Evaluate, Nudge, Summarize Again), a novel summarization method that uses lightweight domain-aware preprocessing to steer summarization into attending more strongly to important information that is commonly lost during the summarization process. We demonstrate that in most domains, summarization errors can be described by a low-dimensional set of categories, and rules based on these categories can be used to guide summarization within that domain to be more resilient against information loss while keeping the same tight summarization budget. We demonstrate SENSA on both technical and non-technical domains, and show it beats SOTA compression on information retention for a wide range of domains and practical compression budgets. SENSA performs especially well on technical fields like coding, where relative improvement in compression information retention can be as high as 100% compared to the next best SOTA method.

Primary area:
foundation or frontier models, including LLMs
Keywords:
summarization, context compression, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BaL6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SENSA, a domain-aware method for improving information retention in LLM-based summarization. It first generates summaries of development documents and uses LLM-generated questions and an LLM judge to identify information lost during compression. These failures are grouped into categories, and the three most common categories are added to the summarization prompt as preservation rules. During inference, documents are split into sections and summarized separately using these rules and proportional token budgets. Experiments show that SENSA generally outperforms existing compression methods, especially on coding tasks.

Critical strengths
  1. The method is simple and interpretable. The learned rules are short and human-readable, are learned once per domain, and need far fewer model calls than baselines.

  2. The experimental setup is thorough and careful. All methods must use 90–100% of the total length budget, so reductive and generative methods are compared at nearly the same length. All prompt examples are also shown in full.

Critical weaknesses

1. Limited validation of the evaluation metric. The paper relies on LLM-generated questions and an LLM judge to measure information retention. Although the authors report high agreement between two judges, this does not establish whether the metric reflects human judgments or downstream task performance. More direct validation would strengthen the evaluation.

2. Unclear practical relevance of the generated questions. The questions are generated from the source documents without considering how the compressed information will be used. It is unclear whether they capture information that matters for real tasks or simply reflect the question generator's preferences. This is particularly concerning for the coding datasets, where retaining more answerable questions does not necessarily mean better agent performance.

3. Potential bias in the evaluation pipeline. The question generator and summarizer use the same model, while the primary judge belongs to the same model family. Although the authors evaluate agreement with a second judge, they do not examine whether using a different question generator would change the results or the ranking of compression methods.

4. Unjustified assumptions in section-wise compression. SENSA allocates token budgets to sections based on their lengths and applies the same preservation rules to every section. However, longer sections are not necessarily more important, and different sections may contain different types of information worth preserving. The paper does not explain why these choices are appropriate or compare them with alternative budget allocation and section-specific compression strategies. This raises concerns about how well the approach would work in real-world tasks.

Questions for authors
  • Do the authors have any evidence that the QA-based information retention metric correlates with human judgments or downstream task performance? This would help clarify whether the reported improvements translate into practical benefits.

  • How sensitive are the results to the choice of question generator? Would using a model from a different family change the generated questions or the relative performance of the compression methods?

  • What proportion of summarization failures is covered by the top three categories (shown in Fig. 3) in each domain? How does performance change as the number of preservation rules increases? These results would help assess the paper's central claim that summarization errors have a low-dimensional structure.

  • What is the practical motivation for the QA-based benchmark used in this paper? Why should performance on automatically generated questions be considered a meaningful measure of context compression quality? In particular, how does this benchmark relate to the information needs of real users or downstream agents?

Reviewer_LE7wOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SENSA (Summarize, Evaluate, Nudge, Summarize Again), a domain-aware text compression method that improves information retention under strict token budgets. It identifies information lost during summarization, groups recurring errors into a small set of interpretable categories, and converts them into rules that guide subsequent summarization. Experiments on four datasets covering legal judgments, government reports, and software engineering show that SENSA generally outperforms existing methods, particularly on coding tasks, where preserving details such as file paths, literal values, and control-flow conditions is crucial.

Critical strengths
  • The proposed method is simple, practical, and interpretable. The learned categories (file or path locations, literal values, identifiers, quantities) are human-readable and reusable across budgets.
  • The proposed SENSA can preserve improved information under tight token budgets.
  • SENSA is efficient in rule generation. The paper reports using 80–90% fewer model calls than ACON to learn preservation rules, suggesting a more cost-efficient approach to improving summarization quality.
Critical weaknesses
  • The novelty is limited: It is close to existing prompt optimization. The method is failure-driven prompt refinement with a clustering step, i.e., categorizing failed probes and adding k = 3 rules. This is close to ACON, LongSumEval’s QA-guided refinement, and general error-driven prompt optimization.
  • Some important ablations are missing:
    • Since SENSA selects the top three rules, experiments with different numbers of rules would help establish whether this choice is optimal or dataset-dependent.
    • There is no section-wise plain summary without rules, so the contribution of the learned rules is never measured properly.
  • Experiments, especially evaluations, need improvement.
    • Missing generic-rule baseline: A fixed generic prompt (“preserve names, numbers, dates, paths and literals”) is needed to test whether domain-specific induction matters at all.
    • There is no measure such as agent success on SWE-bench after compaction, even though ACON, the main comparison point, is defined by downstream agent success.
  • Some important baselines are missing
    • Missing strong baselines including 1) extractive sentence selection, which Jha et al. (2024), cited in the paper, 2) trained abstractive compressors such as Cmprsr, and 3) standard prompt optimizers such as APE, OPRO, and DSPy/MIPRO.
Questions for authors
  • More ablation studies should be conducted to analyze the selection of k, rules, prompts, etc. (see above).
  • More baselines should be compared (see above).
Reviewer_xov3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SENSA. This method is basically a training-free, prompt-level method for compressing(summarizing) text with an LLM. The key idea is to find out what kind of information a naive LLM summary tneds to lose in a particular domain, and to use that knowledge to improve the summarization prompt for the same domain. Concretly, the lost information is identified with LLM-generated probe questions, distilled into a small set of rules, and appended to the summarization prompt.

Performance is measured as how much of the original information remains answerable from the compressed texts across a range of compression budget. The authors claim that SENSA improves information retention over a plain summary and two existing compression methods across four domains (legal documents, government reports, and two sets of coding-agent transcripts), with the largest gain on coding data.

Critical strengths
  1. The proposed method is simple and interpretable. Three human-readable rules are easy to inspect.
  2. Judge robustness check is interesting that re-scoring with a second judge from a different model family yields high agreement with the original judge. This may be useful more broadly for work that relies on LLM-as-a-Judge.
Critical weaknesses
  1. The paper argues that SENSA preserves important information, but what it actually does is much narrower than "preserving important information", which is what the task of summarization is about. The evaluation metric the paper uses is essentially "what a summary addresses (preserves)", and whether the preserved information is actually important is not validated. Information that a plain summary often drops is not necessarily important. This paper never actually answers this important question: is the information in the final summary generated with SENSA actually more important than that in the summaries generated by the baselines? This central claim is not verified.

  2. It seems that the scoring prompt receives only the compressed text and the question, never the correct answer. This basically means that any factually incorrect statement, as long as it is mentioned in the summary, is still scored positively. Wrong facts are never penalized. In the task of summarization, factual correctness is probably one of the most important attributes that any summary should prioritize. This evaluation setup seems to completely miss it.

  3. The answerability score is used as a proxy for summay quality, but this proxy is never actullay validated with human experts. No experiment shows that the answerability score tracks human judgement of quality.

  4. This paper never checks whether the proposed method works better after compression. Without this we never know if this particular compression technique is vaulable after all. It seemes the authors acknowledge this in the limitation section, but this is something to be addressed in the current work, not in future work.

  5. Improtantly, the one baseline that would test the actual contribution, zero-shot rules, is missing. For example, ask the same LLM, in a single call, which three kinds of information are typically important but could be oftren dropped when summarizing documents of this domain. Without this, it is hard to tell that the entire SENSA pipeline add anything beyond what an LLM already knows.

  6. This paper claims human-readable insights as a central contribution, but this claim is never actually validated in practice. For the rules, no domain expert was asked whether the three learned kinds captures what matters in that domain. See Table 6, for example, it seems the kinds are nearly identical for legal and government texts, which would be obvious to any practitioner in these domains. So the idea of providing human-readable insight overstated.

Overall, this paper overclaims relative to what it actually demonstrates, and the evaluation setup has several crucial limitations that make it hard to trust the claimed contributions.

Questions for authors
  1. In chapter 3.2, the paper says "The questions generated for each document are split into a development set and a test set". What does it exactly mean? It sounds like the rules are learned from the same documents that are later used for scoring, which would mean train/test contamination. Please make such descriptions clear.

  2. It seems SENSA gets a lot of score gain from sectioning. Do other baselines also get this advantage?

  3. What happens if the number of rules change?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.