acceptodds
Under review as a conference paper at ICLR 2027

Record Grouping Controls Evidence Weight in Language Models

Abstract

Repeated retrieval records can make one evidence source appear to provide several independent contributions. We group records before generation, remove exact within-group copies, and preserve complementary content in one record per supplied group. In the central natural-text experiment, replacing four falsely split records with one full-content group reduces the normalized likelihood share of the attacker-favored candidate by 18.2–40.7% across all eight model–dataset pairs, while retaining the same 160 evidence words. A stricter six-slot control holds headers, object count, and token length fixed and yields reductions of 1.2–15.6%. On 120 real questions with overlapping distractor excerpts, estimated document grouping improves answer accuracy by 5.42 and 5.83 percentage points for Mistral-7B and Phi-4.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Evidence Grouping, Evidence Provenance, Record Multiplicity, Evidence Weighting, Source Dependence

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_CGDsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether presenting dependent evidence as multiple retrieval records inflates its influence on language-model decisions. It proposes grouping records before generation, removing exact within-group duplicates, and retaining complementary content in one record per supplied group. The paper formalizes invariant and faithful group–content representations and derives partition-error bounds for a bounded external aggregator. Its experiments include content-fixed grouping interventions, matched-serialization controls, and generated-answer evaluation under controlled distractor proliferation.

Critical strengths
  1. Clear and relevant motivation. Distinguishing the number of retrieved records from the number of underlying evidence sources addresses an important evidence-accounting issue in retrieval-augmented generation.

  2. Evaluation across multiple models with useful controls. The study evaluates six checkpoints and includes a stricter control matching record fields and input length. Reporting model- and order-dependent reversals is also valuable and helps expose the limits of the intervention.

Critical weaknesses
  1. The presentation introduces substantial terminology before establishing sufficient intuition, making it very hard to understand. The contribution list introduces invariant and faithful states, content-aware error bounds, and bounded evidence states in quick succession. A running example would help explain why these concepts are necessary and distinguish representation-level properties from guarantees about actual language-model behavior.

  2. The incremental novelty needs a sharper justification. The operational intervention—grouping, exact deduplication, and concatenation—is straightforward. The paper's own related-work discussion describes earlier work on repetition sensitivity, provenance-aware aggregation, and copy-cluster discounting. Simplicity is not itself a weakness, and the content-fixed experimental design is a meaningful distinction. However, the paper should establish more precisely what is new in the method and representation, beyond the experimental isolation of record-layout effects.

  3. The practical significance of the theory is insufficiently demonstrated. The representation characterization relies on quotient factorization, while the partition-error bound follows from bounded group contributions and the triangle inequality. More importantly, the numerical guarantees require a bounded external aggregator and a verified downstream regularity constant, whereas the text experiments enforce one-record-per-group emission rather than these numerical assumptions. The theory therefore does not directly guarantee the observed LLM behavior. I would like to see how it guides an implementable design decision or provides a useful, empirically validated prediction.

  4. The empirical analysis is extensive but remains limited in explanatory and actionable insight. The paper does decompose content retention, false splitting, false merging, and serialization effects, so it would be inaccurate to characterize it as merely reporting aggregate improvements. Nevertheless, the substantial model- and order-dependent reversals leave an important question unresolved: when should grouping help, and when might it hurt? The discussion primarily establishes that presentation matters; a more predictive account or concrete deployment guidance would strengthen the contribution.

Questions for authors

none

Reviewer_qVtFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how RAG models over-weight a source that appears as multiple retrieved records, such as duplicates. To combat this, the authors propose grouping records by source before generation, removing within-group duplicates while keeping complementary content, and presenting each source as a single record. They provide a theory characterizing duplication-invariant representations and bounding the effect of grouping errors. Experiments on small open-weight models show that grouping reduces the influence of over-represented sources, though effects shrink under stricter controls and vary across models. A small HotpotQA study suggests modest accuracy gains.

Critical strengths

[S1] Solving the problem of overweighting duplicated or coordinated sources in RAG is important.

[S2] The experiments are constructed in a controlled manner.

Critical weaknesses

[W1] The paper is unclear and poorly organized. What I understand to be fairly simple ideas (i.e., group passages by source, deduplicate, present one record per source) are wrapped in superfluous, non-standard terminology such as “claim-relative evidence roots,” “canonical evidence quotient,” and “opaque membership labels.” Beyond that, the method and experimental setup are never stated clearly in one place, so understanding the paper is a challenge.

[W2] The method is kind of trival. The authors assume some other process already supplies which retrieved records share a source, then shows that merging those records into a single retrieved record changes model behavior. Despite the theory, this merge step is kind of trivial. In a real-world setting dealing with potential adversaries, identifying the source structure is the actual problem. While the authors do provide some experiments where the source structure isn't assumed in later experiments, it's rather toy settings and they only use off-the-shelf grouping methods.

Questions for authors

None.

Reviewer_PVWcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether presenting passages as separate records makes a language model give them more weight, even when they come from the same source. The proposed approach groups passages by source (removing exact duplicates within each group and keeping distinct content together). Their experiments show that changing record boundaries can substantially change answer preferences without changing the evidence words. A smaller HotpotQA evaluation also finds accuracy gains when grouping overlapping distractor excerpts.

Critical strengths
  • The problem is relevant for RAG performance
  • It is interesting to see a single source gain influence without providing any new evidence. The controlled experiments are designed well and provide useful evidence that record boundaries themselves matter.
Critical weaknesses
  • The same text gaining more weight when split into separate records is an interesting finding. The proposed fix is less convincing. Grouping has smaller effects under stricter controls. It sometimes has the opposite effect across models or passage orders.
  • The proposed intervention is simple: combine passages by source before generation. Given prior work on repetition bias, I am unsure whether the contribution is substantial enough for ICLR.
  • The source-grouping tests are limited. Section 5.2 uses fictional records from a known campaign brief. HotpotQA uses passages from one document. A harder case is several news articles based on one press release. Can the method identify their shared origin and group them correctly?
Questions for authors
  1. On HotpotQA, does MiniLM grouping outperform random grouping when each group’s size is kept the same? This would help show whether finding the right groups matters.
  2. Have you tested real articles that share a2 source, such as several news stories based on one press release? Can the method identify their shared origin, and does grouping them improve answers?
  3. Why does grouping increase the campaign-favored answer’s probability for Mistral in Section 5.2? Does this happen across most product pairs and in both passage orders, or is the average driven by a few large changes?
Reviewer_FEmdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how in large language models, splitting retrieved records across different groupings changes the importance assigned to a piece of evidence. Based on this they develop a defense which uses a supplied partition specifying which content contributes together, grouping those records to prevent excessive importance from being put on them while also maintaining distinction between groups. They validate this on four open-weight models where grouping the same 160 evidence words reduces preference for the manipulated answer by 18–41%, and on 120 HotpotQa questions where estimated grouping improves answer accuracy by ~5 points.

Critical strengths
  1. The results from prompt level deduplication seem very strong since it looks like the model cannot recover despite knowledge of the duplication, and structural deduplication gets better results which points to the strength and utility of the defense.
  2. The test sample size is rigorous and convincing, they hold the ordered 160 evidence words fixed and vary the record placement for their controls.
Critical weaknesses
  1. There are no details on what mechanism causes this behavior and no model internals observations, which should not be hard to get since the models used here are all small open-weight models (the largest being 14B parameters).
  2. It would be very useful to see larger models compared in this analysis, since some of the results may not transfer. Especially the result where telling the model about the repeated evidence does not improve performance feels like it could be an artifact of small model size.
Questions for authors
  1. Would it be possible to test on bigger open source models? Even closed source frontier models could be tested on the controlled generation experiments in appendix C
  2. (Suggestion) A mechanistic study of attention heads or other model internals could help the paper significantly, and possibly lead to a better intervention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.