acceptodds
Withdrawn from ICLR 2027

The Rich Get Relevant: Neuro-Symbolic Diversity-Inducing Ranking on Semantic Graphs for Personalized Multi-Document Summarization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Query-focused multi-document summarization (MDS) underpins many generative-AI applications (automatic presentations, video and meeting summarization, descriptive question answering, personalized news digests). Yet feeding a long-context LLM raw multi-source text is unreliable: it triggers "lost-in-the-middle" behavior, cross-source redundancy, and hallucination, while classical graph rankers are "feature-blind" and cannot personalize to user intent. We propose RICH, a lightweight (10^5–10^6-parameter) neuro-symbolic ranker that hands the LLM a structured ranked list of (topic, evidence, score) tuples instead of raw text. RICH extracts intent-filtered atomic topics, organizes them in a hierarchical topic–document graph, and runs a parameter-free Recurrent GNN with intent-aware MLP priors and Graph-Attention transitions that scores each topic by jointly balancing relevance and diversity. It embeds the discrete DivRank random walk inside a differentiable model and offers three simple guarantees: RICH is a differentiable form of Pointwise DivRank; its recurrence is a stable damped walk on the simplex with a score floor and a fixed point (fast convergence measured on all evaluation graphs); and its hierarchical prior gives every source document a positive score floor. Across four DiverseSumm setups and an event-level held-out split with disjoint train/test events, RICH leads strong LLM-prompting and classical content-selection baselines on coverage and diversity, with the largest and statistically significant gains in the query-focused settings. The gains transfer zero-shot to Multi-News and WCEP, and a human study confirms the reliability of the evaluation protocol. RICH adds negligible inference overhead ( 0.3% of end-to-end latency) on top of the LLM calls that every baseline also makes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Query-focused multi-document summarization, Diversity-aware content selection, recurrent graph neural networks, Neuro-symbolic graph ranking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PHWvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RICH, a lightweight neuro-symbolic ranker for query-focused multi-document summarization. Source documents are decomposed into atomic topics arranged in a hierarchical topic-document graph; intent-aware MLP priors and graph-attention kernels feed a parameter-free recurrent solver that unrolls the DivRank vertex-reinforced walk, trained with self-supervised coverage, determinantal point process, and sparsity objectives.

Critical strengths
  • The core idea is clean and well executed: unrolling a classical diversity-inducing walk as a differentiable layer keeps the symbolic routing intact while making the whole ranker trainable without labels.
  • The evaluation is more careful than typical for this area: an event-level held-out split with disjoint events, zero-shot transfer to two corpora, significance tests in the query-focused settings, a human study of the judge with agreement statistics, and honest reporting of where baselines win.
Critical weaknesses
  • The evaluation data, the training intents, and the judge all come from the same model family: GPT-5.1 generates the augmented questions and the synthetic intents and also judges Coverage and Answerability. The human study validates overall judge accuracy on a sample but does not test whether the judge systematically favors the output format of one system, and RICH feeds the judge a structured ranked format while baselines do not.
  • The component ablations rest on 30 events where most pairwise differences are not significant, so the evidence for the contributions of individual terms is directional rather than confirmed.
Questions for authors
  • Could you validate the GPT-5.1 judge with a second judge family, or have humans score a stratified sample across all four systems?
  • Which events supplied the validation loss for early stopping, and are they disjoint from the 49 held-out events? If not, the held-out generalization result is compromised.
Reviewer_xtXCOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes RICH\texttt{RICH}, a neuro-symbolic framework for query-conditioned multi-document summarization. It aims to improve the relevance, coverage, and diversity of the information spanning multiple documents. It first extracts atomic facts from documents and converts to a semantic Knowledge Graph that connects topics and their source documents. It then combines intent-driven neural priors, GATs, and a differentiable recurrent formulation (DivRank) that learns topic weightage and discourages redundant content. The experiments evaluate generic query-focused MDS and shows improvement in coverage and semantic diversity.

Critical strengths

For critical strengths, my rating is NONE\textbf{NONE}.

Although there are acceptable strengths such as differentiable diversity-aware ranking and structured evidence-guided summarization, they do not override the core problem formulation which is stated in the weaknesses.

Critical weaknesses

W1: Fundamental mismatch between the claimed Personalization problem and implemented Query-Focused MDS (Primary Blocker)

The central concern is that the authors position their contribution as personalized MDS, whereas the actual problem formulation addresses conventional query-focused MDS. The work considers the user query or intent itself as the personalization interface. However, conditioning content selection solely on a query does not establish adaptation to the subjective information needs of different users. The proposed ranking depends on the source documents and the query, without considering user-specific context, preferences, or individual information needs beyond the given query.

I feel this distinction is particularly important and decision-critical because the relevance and importance of information can be inherently subjective and user-dependent, even when users express identical queries.

For example, consider two users reading the same collection of news articles about an earthquake and asking exactly the same question: "Summarize the economic impact of the earthquake."

  • Alice, a local small-business owner, may prioritize business closures, financial assistance, reopening costs, and local employment consequences.
  • Bob, a supply-chain analyst, may prioritize manufacturing disruptions, transportation bottlenecks, production losses, and broader economic consequences.

Although the documents and queries are identical, their subjective information priorities can differ substantially. Under the proposed formulation, both users receive the same topic ranking because their individual contexts are not represented. This makes it unclear what exactly is being personalized beyond explicit query conditioning.

The experimental design also reinforces this concern. The authors evaluate generic and query-focused summarization using synthetically generated intents, where the metrics focus on coverage, diversity, and answerability. These are appropriate measurements for QF-MDS, but they do not establish whether the framework can address different subjective information priorities across users. Therefore, neither the problem formulation nor the experimental results convincingly justify the claimed personalization contribution.

W2: Limited methodological novelty and lack of sufficiently new empirical insights for QF-MDS (Decision-critical)

Even if the work is considered purely as a QF-MDS contribution, I find the methodological novelty and empirical insights are not sufficiently convincing.

The overall approach of extracting atomic facts, selecting relevant and diverse information, and providing selected evidence to an LLM has already been explored in "Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries" (Padmakumar et al., ACL 2025). Similarly, the idea of balancing information importance and diversity through reinforced random walks was introduced in "DivRank: The Interplay of Prestige and Diversity in Information Networks" (Mei et al., KDD 2010), which also considered summarization applications.

The proposed differentiable recurrent DivRank, combined with learned neural priors and hierarchical document-topic interactions, is an interesting architectural modification. However, I am not very convinced that this integration brings a sufficiently distinct methodological advancement over these established ideas. The experiments do not include a direct comparison with conventional DivRank-based content selection in the same setup. This makes it difficult to understand how much improvement actually comes from the proposed differentiable recurrent formulation, rather than the already established relevance-diversity ranking principles.

The empirical findings also largely follow observations which have been established earlier: explicit content selection improves multi-document coverage, while jointly considering relevance and diversity helps selecting more informative and non-redundant content. Although the reported results show improvements over the considered baselines, including held-out and zero-shot evaluations, I do not find a substantially new insight about the QF-MDS problem emerging from these results.

Therefore, beyond the fundamental personalization concern, the independent contribution to QF-MDS also appears insufficiently differentiated from prior research, both in its methodology and empirical findings. This makes the overall significance of the proposed work difficult to justify for acceptance.

Questions for authors
  1. Is there any mechanism that captures user-specific information needs beyond query conditioning, and what fundamentally distinguishes this formulation from conventional QF-MDS?

  2. Can the authors identify experimental evidence demonstrating adaptation to different subjective user priorities under identical documents and queries?

  3. What distinct methodological advantage does the differentiable recurrent DivRank provide over existing relevance-diversity ranking methods, and is there direct experimental evidence supporting this advantage?

1 comment
Reviewer_xtXCOpenReview ↗

Invitation for Further Discussion on Key Concerns

Dear Authors,

I would be interested to hear your perspective on the main concerns raised in my review, particularly regarding the personalization formulation and the independent methodological contribution. Since the discussion period is approaching its end, I thought an earlier response, if possible, would allow me to have a more meaningful exchange rather than only a last-minute assessment.

I remain open to reconsidering my evaluation based on your clarifications and supporting evidence.

Reviewer_jFexOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The papers proposes RICH, a content selection method for query-focused multi-document summarization. The method first employs a LLM to extract key points from source documents, then constructs two interconnected graphs: one representing the key points and the other representing the documents from which they are derived. A learned ranking procedure selects relevant and diverse content, which is subsequently provided to an LLM to generate the summary. Experiments compare this approach against baselines based on prompting and content-selection techniques, including an evaluation on events not encountered during training, as well as an assessment of its transferability to other datasets.

Critical strengths

I find the use of two connected graphs interesting. Modeling both the key points and their source documents provides a useful way to consider relevance and redundancy while keeping track of where the selected information comes from. Although separating content selection from summary generation is already common in the literature, it remains practically useful because the ranking module can be combined with different generators.

Critical weaknesses

1) The contribution appears incremental, and its novelty is not sufficiently established against closely related methods.

I recognize that RICH introduces a specific ranking method with new design choices. My concern is therefore not that the method contains nothing new, but that the contribution remains narrowly focused on selecting content more effectively within an established extract-then-generate pipeline. Many existing MDS approaches follow this general strategy, each proposing a different way to identify relevant information and reduce redundancy. The paper needs to explain more clearly what readers gain from RICH beyond another content-selection method with improved scores.

The two-level graph design is interesting, but its significance is difficult to assess without a more complete discussion of closely related methods. The related work is too brief and mainly focuses on classical graph rankers, leaving out more recent graph-based extract-then-abstract approaches for MDS. In particular, G-SEEK ("Graph-Based Abstractive Summarization of Extracted Essential Knowledge for Low-Resource Scenarios", ECAI 2023) and G-SEEK-2 ("Cross-Document Distillation via Graph-based Summarization of Extracted Essential Knowledge", IEEE/ACM TASLP 2025) use heterogeneous input graphs to identify salient information before passing it to a generative model. These works are directly relevant to the graph-based content-selection and long-context discussions in MDS. The authors should explain how RICH differs in its graph construction, learning procedure, and selection behavior, and why these differences represent a meaningful advance. Graph-based preprocessing and compatibility with different generators are already established ideas. The discussion should also cover complementary approaches to combining information across documents. For example, DAMEN ("Discriminative Marginalized Probabilistic Neural Method for Multi-Document Summarization of Medical Literature", ACL 2022) generates summaries through token-probability marginalization, while RAMSES ("Retrieve-and-Rank End-to-End Summarization of Biomedical Studies", SISAP 2023) jointly learns retrieval and generation using this approach. These methods differ from RICH, but discussing them would help readers understand the alternatives for selecting and combining useful information without simply concatenating all source documents.

The comparisons with LLM-KPs and LLM+DPP are informative, but they do not fully establish the contribution relative to this broader literature. I would like the paper to identify its main advance more directly: is it the coupling between the two graphs, the learned recurrent ranking procedure, or the training objective? The experiments and discussion should then explain what that advance enables and what we learn from it. As currently presented, I see an incremental improvement in content selection, while the case for a substantial methodological contribution remains insufficiently developed.

2) The writing feels heavily AI-generated and makes the paper unnecessarily difficult to follow.

Much of the writing reads as heavily AI-generated to me: long sentences packed with technical terms, frequent semicolons, and compressed statements that sound polished but require substantial effort to unpack. My concern is the effect on readability, rather than the use of writing tools itself. Using AI to improve the writing is perfectly acceptable, but the authors should remember that the paper is meant to be read and understood by humans, not by AI. It should help readers understand the ideas and take away a clear message, whereas the current presentation often makes this harder than necessary.

For example, lines 63–73 introduce almost the entire architecture before explaining the basic purpose of its components in simple terms. The contributions paragraph, related work, and discussion of the results have the same problem.

I also question whether the theorems need to appear in the main paper. Their practical role is not sufficiently clear, and the formal statements make the presentation heavier without helping me understand what they imply for the final summary. I would prefer a plain-language explanation of the guarantees in the main text, with the formal statements and proofs in the appendix. If the theorems are central to the contribution, please explain more directly why each is needed and what readers learn from it.

I would recommend a substantial rewrite throughout the paper, using shorter sentences, fewer compressed technical descriptions, and clearer explanations. Technical detail is welcome, but readers should not have to reconstruct the main idea from a sequence of densely packed terms.

Questions for authors
  • Regarding topic extraction in stage S1, how much does the final result depend on the extractor? The experiments change the extractor and generator together, so it is difficult to separate their effects. Have you tried changing only the extractor while keeping the generator fixed? I would like to understand whether extraction quality is a major bottleneck.

  • How were the extracted topics validated? In particular, did you check whether they are supported by the source, sufficiently atomic, and complete enough to preserve important information? The cosine filter checks alignment with the user intent, but relevance to the intent does not by itself establish factual support.

  • Appendix G provides useful timing information and shows that extraction takes a substantial share of the total runtime. Could you clarify how the cost and quality change with different extractors, and which parts can be reused when the user intent changes?

  • Regarding the human study in Appendix I, could you provide the complete annotation instructions and explain how the materials were presented? For example, was a web interface or another annotation tool used? Please also clarify the annotators' relevant experience, whether participation was compensated or voluntary, and how labels were combined for items with multiple annotations.

  • The experiments use Llama-3.1-8B and GPT-4o, which are relatively old models. The paper already shows smaller gains with GPT-4o and attributes this to its stronger ability to process the input. This raises the question of whether RICH still provides a meaningful improvement with newer, more capable models. Have you tested this? Such a comparison would help establish whether the benefits remain relevant as the generation backbone improves.

  • Why was ROUGE omitted entirely? I understand that lexical overlap does not fully capture coverage, diversity, or factual consistency. However, on datasets with suitable reference summaries, reporting ROUGE alongside the other metrics would help connect the results to established summarization evaluations and show whether the metrics lead to similar conclusions.

  • In Figure 1, explicitly explain that (v_1, ..., v_6) are extracted topic nodes and show how they correspond to source documents. Please also improve spacing: the graph labels in S2 extend beyond the box, the upper blue curve is close to the heading, and the labels and equations in S3–S4 are crowded.

  • Add a short introduction to Appendix E explaining the notation table and place it close to the section heading.

  • Rewrite the contribution list as a small number of clearly separated points. Readers should be able to identify the main contribution without first unpacking every technical component.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.