- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes OriginCount, a provenance-aware evidence aggregation method for retrieval agents. The key motivation is that multiple retrieved pages may originate from the same source, causing agents to overcount dependent evidence and become overconfident. OriginCount groups pages using citations, shared identifiers, and restated content, assigning one vote per provenance group. The authors provide theoretical analysis and evaluate the method across six web snapshots, five language model backbones, and three retrievers.
Overall, the paper addresses a meaningful problem and presents a simple, interpretable approach with promising empirical results.
Critical strengths
-
The paper identifies an important problem in retrieval agents that multiple retrieved pages do not necessarily represent independent evidence. Distinguishing information provenance from document similarity is meaningful and practically relevant.
-
OriginCount provides a simple and interpretable solution without requiring additional model training. The theoretical analysis offers a principled justification for the proposed counting rule under explicit assumptions.
-
The experiments cover multiple language model backbones, retrievers, and agent variants. The ablation studies further investigate provenance signals, grouping strategies, and missing provenance information, providing useful evidence for the effectiveness of the method.
Critical weaknesses
-
The construction and annotation of the six web snapshots are insufficiently described. More detailed justification is needed to establish the reliability and representativeness of these benchmarks.
-
The stopping evaluation relies on prefetched retrieval calls rather than an online retrieval process. Furthermore, Appendix C.1 calculates the mean stopping round only over questions that reach the stopping threshold. Given that the default threshold requires at least seven supporting origins, while Table 35 reports a median of four origins among the sampled questions, it is unclear how frequently OriginCount actually stops early.
-
The reduction in CE rate comes with a decrease in answer accuracy from 72.98% to 71.99%. . Although the paper includes a flat-discount baseline, a more detailed analysis of the accuracy–calibration trade-off would help clarify the practical benefits of OriginCount.
Questions for authors
-
How were the ground-truth provenance labels constructed and validated? Is the approximately 80% copy rate representative of real-world web retrieval?
-
What proportion of questions reaches the stopping threshold under OriginCount, and how many exhaust the retrieval budget?
-
Can the authors provide additional calibration and selective-prediction evaluations, such as risk–coverage curves or Brier scores, to clarify whether OriginCount improves answer reliability beyond producing more conservative confidence estimates?
I appreciate the rebuttal. It has fully resolved my concerns. I have adjusted my rating