acceptodds
Under review as a conference paper at ICLR 2027

Probability-Weighted Pooling for Unified Context Compression and Reranking

Abstract

Retrieval-augmented generation (RAG) systems commonly use separate models for context compression and document reranking, requiring redundant encoding of the same query-document pairs. Unifying these stages allows a shared encoder to rank documents and prune the contents in a single pass. However, unified ranking requires choices in how to pool token representations and supervise the resulting ranking score. In this work, we introduce probability-weighted pooling, which weights document-token representations by the relevance probabilities predicted by the compression head. We theoretically prove that, as the predicted relevance weights in probability-weighted pooling become more accurate, its ranking score converges to that obtained using ground-truth evidence weights. We conduct comprehensive experiments across six benchmarks, comparing three pooling mechanisms and two ranking targets. Although the average improvement is modest, the gains are more pronounced on PubMed, where queries often involve multiple relevant documents, consistent with the statistically significant advantage observed in our aggregate comparisons. Document-level supervision yields a larger improvement over span-or-penalty supervision. These results highlight the greater impact of ranking supervision relative to pooling choice.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Retrieval-augmented generation (RAG), context compression, document reranking, probability-weighted pooling, ranking supervision, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_x8KtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The authors propose to unify context compression with reranking in RAG systems via a shared encoder and weighted pooling.

Critical strengths

None

Critical weaknesses
  • The paper was poorly written and difficult/impossible to fully understand. It lacks proper technical context/background, clear definitions of key terms/symbols, or explanatory diagrams.
  • The paper seems highly related to reranking. While there is an abundance of related work in the literature about reranking, the paper ignores it all.
Questions for authors

N/A

1 comment

Thanks for the revision, which helped provide more context and clarify additional details, such as defining some previous undefined symbols. As of the latest revision, the contribution of the work seems marginal as stated in the new abstract and conclusion.

Reviewer_Zz4vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the ranking head in a model that jointly compresses retrieved passages and reranks them for RAG. Its main proposal is to pool document-token representations using the evidence probabilities already produced by the compression head. The experiments compare this approach with mean and CLS pooling, and compare full-document teacher scores with span-or-penalty supervision. On six QA datasets, the larger empirical difference comes from the supervision target; the pooling differences are considerably smaller. The theoretical analysis establishes convergence to an internal oracle-pooled score as token prediction loss vanishes, with the representations and readout held fixed.

Critical strengths
  1. Reusing the compression probabilities for document pooling is a simple, well-motivated design that fits the shared-encoder setting without another encoding pass. The comparison with uniform mean pooling is particularly useful because it isolates the effect of the probability weights within the same implementation. The consistency result provides a precise account of what accurate evidence predictions imply for this pooling operator, and the paper explicitly distinguishes that result from a guarantee about teacher scores or downstream accuracy.

  2. The evaluation covers six QA datasets and a useful range of pruning thresholds and retention counts. The difference between document-target and span-or-penalty training is substantial in these experiments: probability-weighted DOC is ahead by 3.71 accuracy points on average (Table 2), with a positive dataset-averaged difference on all six datasets. The attention to actual prompt size, dependence between evaluation conditions, and metric sensitivity makes the study more informative than a comparison at a single operating point.

Critical weaknesses
  1. The evidence is not yet strong enough to recommend the proposed pooling rule over the existing alternatives. Its mean advantage over uniform pooling is 0.26 accuracy points, and the reported significance depends on an ICC estimate obtained from only six datasets. Table 8 shows that this comparison is no longer significant at ICC = 0.10, while Table 7 gives an ICC interval extending to 0.488. There is also only one trained checkpoint per variant. The probability-weighted head does not significantly outperform CLS pooling (Table 2), or either document-target alternative on the accuracy-prompt-size hypervolume (Table 10). Appendix J further shows that its advantage over mean pooling is specific to the SOLAR judge among the reported metrics. The authors appropriately acknowledge these limitations. With the present evidence, however, I cannot tell whether the small observed gain would persist across training runs and a check of the judge's decisions. This matters because the pooling rule is the principal methodological contribution.

  2. The larger supervision result has a narrower interpretation than the proposed explanation suggests. DOC already uses the document-score target adopted by Provence. Its comparison with span-or-penalty changes both the teacher's input on evidence-positive examples and the labels for examples without a supplied span, which are replaced by a constant penalty. The span scores also have a different distribution, and training uses pointwise MSE in a joint objective. Consequently, the performance gap can reflect loss scaling or the treatment of no-span examples as well as missing context. The score histograms in Appendix I/Figure 5 do not distinguish these possibilities. The theoretical result does not resolve this issue: it concerns a fixed model's internal reference on evidence-positive blocks and does not establish which supervision target yields better rankings.

I lean toward weak rejection. The design is sensible and the empirical observations are useful, but the current evidence leaves the benefit of the new pooling rule uncertain and does not sufficiently explain the stronger supervision effect. A more reliable pooling comparison or a controlled account of the supervision gap would materially improve my assessment.

Questions for authors
  1. For probability-weighted versus mean pooling, could you provide the six dataset-averaged accuracy differences and the corresponding dataset-level paired analysis? Is there any repeated-training evidence that the difference is larger than seed variation? A limited check at a fixed pruning threshold and retention count would be more informative here than extending the operating-point sweep.

  2. Can you separate the effect of changing the teacher input from the effect of assigning a constant target to no-span examples? In particular, does the DOC advantage persist when no-span examples receive the same targets in both conditions and the ranking-loss scales are made comparable? This would help determine whether decontextualization explains the result.

  3. Could you validate a small, blinded sample of the queries on which probability-weighted and mean pooling receive different SOLAR judgments, using human assessment or an independent judge? A focused check of these disagreement cases would help establish whether the observed advantage reflects answer correctness.

Reviewer_2oifOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes probability-weighted pooling for the ranking head of a unified compression and reranking model built on Provence. The pooling uses the compression head’s token probabilities as weights and is compared with CLS and mean pooling, alongside two ranking targets, the teacher’s full-document score and a span-or-penalty target. The paper also gives a theoretical analysis of the proposed pooling. On six QA datasets, the choice of target has a substantial effect while the choice of pooling has a negligible one.

Critical strengths
  • The appendix accounts for correlation across datasets in its statistical tests.
Critical weaknesses
  • The proposed method has no advantage over the existing default, and the contribution would remain marginal even if additional seeds confirmed the small gain over mean pooling. The remaining finding, that the target matters more than the pooling, is not sufficient for a paper.
Questions for authors
  • If the three heads are retrained with several seeds, does probability-weighted pooling consistently outperform CLS pooling?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.