acceptodds
Under review as a conference paper at ICLR 2027

OriginCount: Counting Origin Events Instead of Pages in Retrieval Agents

Abstract

A retrieval agent answers a question by fetching pages, counting how many of them support a candidate answer, and stopping once the count is high enough. That count is usually a count of pages. On the web many pages restate one original record, a wire report, a press release, or a dataset card, without adding a new observation; we call such a page an echo of its origin. An echo carries no evidence beyond its origin: if pages are produced from a record and add nothing to it, then , so a page count applies the evidence of once for and once more for every echo. On six web snapshots with page-level provenance labels, of retrieved pages are echoes, and counting by origin instead of by page changes the stopping round or the answer on of the questions. OriginCount groups pages by the origin they can be traced to, gives each group one vote, keeps a page of unknown origin as its own vote, and leaves the content of every page pooled. On a Qwen3-32B host it lowers the rate of confidently wrong answers from to and the calibration error from to , at a cost of one point of accuracy, which the true labels trace to grouping errors, and a rule-based provenance extractor makes it cost ms per page; every comparison is paired by question at one budget of eight prefetched calls. The correction repairs copy inflation and nothing more: it cannot make distinct origins independent, and where copies had propped up a correct answer, grouping them lowers accuracy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Retrieval agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S9VGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OriginCount, a provenance-aware evidence aggregation method for retrieval agents. The key motivation is that multiple retrieved pages may originate from the same source, causing agents to overcount dependent evidence and become overconfident. OriginCount groups pages using citations, shared identifiers, and restated content, assigning one vote per provenance group. The authors provide theoretical analysis and evaluate the method across six web snapshots, five language model backbones, and three retrievers.

Overall, the paper addresses a meaningful problem and presents a simple, interpretable approach with promising empirical results.

Critical strengths
  1. The paper identifies an important problem in retrieval agents that multiple retrieved pages do not necessarily represent independent evidence. Distinguishing information provenance from document similarity is meaningful and practically relevant.

  2. OriginCount provides a simple and interpretable solution without requiring additional model training. The theoretical analysis offers a principled justification for the proposed counting rule under explicit assumptions.

  3. The experiments cover multiple language model backbones, retrievers, and agent variants. The ablation studies further investigate provenance signals, grouping strategies, and missing provenance information, providing useful evidence for the effectiveness of the method.

Critical weaknesses
  1. The construction and annotation of the six web snapshots are insufficiently described. More detailed justification is needed to establish the reliability and representativeness of these benchmarks.

  2. The stopping evaluation relies on prefetched retrieval calls rather than an online retrieval process. Furthermore, Appendix C.1 calculates the mean stopping round only over questions that reach the stopping threshold. Given that the default threshold requires at least seven supporting origins, while Table 35 reports a median of four origins among the sampled questions, it is unclear how frequently OriginCount actually stops early.

  3. The reduction in CE rate comes with a decrease in answer accuracy from 72.98% to 71.99%. . Although the paper includes a flat-discount baseline, a more detailed analysis of the accuracy–calibration trade-off would help clarify the practical benefits of OriginCount.

Questions for authors
  1. How were the ground-truth provenance labels constructed and validated? Is the approximately 80% copy rate representative of real-world web retrieval?

  2. What proportion of questions reaches the stopping threshold under OriginCount, and how many exhaust the retrieval budget?

  3. Can the authors provide additional calibration and selective-prediction evaluations, such as risk–coverage curves or Brier scores, to clarify whether OriginCount improves answer reliability beyond producing more conservative confidence estimates?

Reviewer_4B32OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how retrieval agents can become overconfident when multiple retrieved pages repeat information from the same underlying source. The proposed method, OriginCount, groups pages using three types of provenance signals: explicit references, shared identifiers, and inferred restatements. It then replaces the page-level support count with the sum of mean support within each group, while treating untraced pages as separate groups. The paper provides theoretical results on copy invariance and incomplete provenance and evaluates the method on six web snapshots with provenance labels.

In the main Qwen3-32B setting, OriginCount reduces the confident-error rate (CE) from 16.43% to 11.34% and the expected calibration error (ECE) from 14.93% to 9.08%. Accuracy, however, decreases from 72.98% to 71.99%. The results suggest that accounting for source dependence can reduce overconfident errors, but the evaluation leaves open whether the gains reflect better confidence estimates, reduced high-confidence coverage, or both. The practical benefit for online retrieval also remains unclear because all retrieval calls are prefetched.

Critical strengths
  1. The problem is well motivated. Multiple pages derived from the same report do not necessarily provide independent corroboration. The paper also makes a useful distinction between claim-level provenance and domain-level grouping, since pages from the same domain can report independent information and copies can appear across different domains.

  2. The method is straightforward to integrate. OriginCount changes the counting step without retraining the underlying model or removing pages from its context. Treating untraced pages as separate groups avoids making unsupported provenance assignments. The rule-based extractor also retains part of the reported improvement at substantially lower cost.

  3. The evaluation covers several relevant settings. The authors test five model backbones, three retrievers, multiple agent variants, and a range of baselines. The oracle comparison and ablations help distinguish the potential benefit of provenance-aware counting from the errors introduced by estimated grouping. The reported CE reduction relative to page counting is consistent across the tested backbones and retrievers.

Critical weaknesses
  1. The evaluation does not clearly separate improved reliability from reduced high-confidence coverage. CE is the fraction of all questions answered incorrectly with confidence of at least 0.8, so it depends both on the proportion of questions receiving high confidence and on the error rate within that group. On NEWS-A, the share of all questions assigned to the high-confidence bin falls from 41.4% to 32.0%, while accuracy within that bin rises from 59.6% to 61.7% (Table 8). These results suggest that much of the CE reduction may be attributable to fewer questions receiving high confidence, rather than a large improvement in the reliability of such predictions. The two-bin ECE also provides limited information about calibration within each bin. A comparison at matched confidence coverage, together with risk–coverage curves and more detailed calibration metrics, would make the results more informative.

  2. The stopping-round results do not establish actual retrieval savings. All methods receive eight prefetched retrieval calls and the same fixed set of pages (Appendix C.1). Stopping earlier therefore does not reduce the number of retrieval calls actually executed, nor does the experiment test a setting in which subsequent retrieval calls depend on the agent’s stopping decision.

Questions for authors
  1. Can the authors compare Page Count and OriginCount at matched high-confidence coverage? In particular, how much of the reduction in CE remains when both methods make high-confidence predictions on the same proportion of questions? Risk–coverage curves and more fine-grained calibration metrics would help clarify this point.

  2. Since retrieval calls are prefetched in the current experiments, can the authors evaluate OriginCount in an online retrieval setting where the stopping decision determines whether another search is performed? Please report the fraction of questions that stop early, the fraction that exhaust the retrieval budget, and the resulting retrieval cost and answer accuracy.

  3. How were the provenance labels for the six snapshots constructed and independently validated? Could the authors provide sufficient details of the Level-3 extractor to reproduce its decisions? Additionally, under the "Labels through all three levels" condition in Table 18, adding Level 3 increases the MAE of the estimated number of origin groups (G^\hat{G}) from 0.157 to 0.197, while the main experiment reports a reduction in CE from 13.33% to 11.34%. Which types of grouping errors explain this difference, and can the authors quantify their respective effects on downstream performance?

Reviewer_1aDvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that retrieval agents which count supporting pages double count evidence, since many web pages copy one origin. It proposes OriginCount, which groups pages by citations, shared identifiers, or restated content and gives each group one vote. On six labeled web snapshots with a Qwen3-32B agent, it reports confident errors dropping from 16.4% to 11.3% and ECE from 14.9% to 9.1%, at about one point of accuracy.

Critical strengths

The problem is real and clearly stated, and the fix is simple. It needs no training, only replaces the counting step, and falls back to the page count when no provenance is found. The ablations on grouping unit and untraced pages are also well designed.

Critical weaknesses
  1. The gains look mostly mechanical. OriginCount never gives a higher count than the page count, so confidence is lower across the board. At a fixed 0.8 cutoff, that alone cuts confident errors. Table 8 shows the confident share falling from 41% to 32%, while accuracy in the confident bin barely moves (59.6% to 61.7%). For the page count, the oracle, and OriginCount alike, the confident bin is less accurate than the unsure bin. The paper needs threshold-free metrics such as risk-coverage curves or AURC, and a page-count baseline with a tuned threshold or temperature.

  2. The data cannot be assessed. The paper never explains how the six snapshots were built or how the provenance labels were obtained. The Table 18 caption suggests the labels are traced through the same three signal levels the method uses, which would make the evaluation partly circular. Several supporting checks hold by construction, such as no change without provenance and no wrong merges in synthetic pools. No code is provided.

  3. Several numbers do not add up.

    • Intervals. In Table 1, some intervals match a t-interval over the six snapshots (self-report, flat discount), while others are about half or a fifth as wide (vs. page count, vs. reliability plus de-dup).
    • Stopping. Reaching the 0.9 threshold needs about 7 supporting origins, yet Table 35 reports a median of 4 origins per question. The oracle still stops at round 3.3 on average.
    • Cost. The flat discount, a single multiplication, costs 4.2 ms per page. The full rule-based configuration is cheaper than Level 1 alone and leaves more pages untraced, even though it includes Level 1.
  4. Novelty and theory are limited. Discounting copied sources is the core idea of copy-aware truth discovery. Dong et al. (VLDB 2009) did this, and the truth-discovery literature has also used explicit structure such as retweet graphs rather than only shared claims. Tracing claim provenance with an entailment model, which is essentially level 3, appears in Zhang, Ives and Roth (ACL 2020), and this work is not cited. The paper also does not compare against TrustRAG, which clusters near-duplicate injected documents, or RobustRAG's isolate-then-aggregate voting. On the theory side, Lemma 1 and Proposition 1 are known results. Theorem 1 follows almost directly from its own restriction to equal weights inside a group, and the mean score does not follow from the paper's own Assumption 1.

Questions for authors
  1. How were the snapshots and provenance labels created, and are the labels derived from the same three signals the method uses?
  2. Which procedure produced each confidence interval in Table 1?
  3. What fraction of questions actually stop under each method, given the origin counts in Table 35?
  4. Which LLM is the level-3 extractor, and how does it run at 16 ms per page?
  5. How do results look at matched coverage, or with the page-count threshold tuned on held-out data?

Comments on the paper

2 comments

I appreciate the rebuttal. It has fully resolved my concerns. I have adjusted my rating

AuthorsOpenReview ↗

Title: General Response 2: What the paper finds and what is new, in plain terms

Dear AC and reviewers,

Thank you for the reviews and the discussion. As the discussion closes, we restate in plain terms what the paper finds and what is new.

The problem. A retrieval agent decides how sure it is by counting the pages that support an answer, and it stops searching once the count is high enough. On the web, most pages copy a few original reports: a wire story is rewritten by many sites, and a statistics release is repeated by news sites and blogs. Counting pages then counts the same report many times. It is like asking ten people for directions when nine of them only repeat what the first one said: ten answers, one opinion.

What we find. On six web snapshots, 78% to 82% of the retrieved pages are copies, with 4 to 5 pages per origin on average. The page count is most confident exactly where one origin was copied most, so its confident answers are the copy-inflated ones: it ranks its own answers worse than random (Table R1-A). The usual tools do not fix this. De-duplication removes verbatim copies, but even semantic de-duplication keeps 14% to 35% of rewritten copies, because a rewrite can look as different as an independent report (Proposition 1). Reliability weighting asks how often a site is right, not whether a page is independent.

What is new.

  1. Count origins, not pages. OriginCount links pages that cite each other, share an identifier, or restate the same content, and gives each group of linked pages one vote. A page with no links keeps its own vote, and every page stays in the agent's context.
  2. Know what the count guarantees. Among such counts, one vote per group is the only one that copies cannot change (Theorem 1). With correct links, the count lies between the true number of origins and the page count, so missing provenance only moves it back toward the usual count (Theorem 2).
  3. Use the count where it matters. The count sets the agent's confidence and its decision to stop searching, so it changes what the agent does, not just a score.

What it buys. With a Qwen3-32B agent, confidently wrong answers fall from 16.4% to 11.3% of the questions, and the calibration error from 14.9% to 9.1%, at about one point of accuracy. This is more than lower confidence: at matched coverage OriginCount still makes fewer confident errors, it ranks answers better than any other deployable method, and it changes the final answer on 10% of the questions, which no rescaling of confidence can do. With online search at the same number of calls, it also raises accuracy, because it keeps searching until an independent origin turns up (Table R3-B). It needs no training, holds on five backbones and three retrievers, and adds to robust RAG methods such as RobustRAG.

What is classical, and what is new. That copies should not count as independent evidence is an old idea, from data fusion to truth discovery (Dong et al.), and Section 2 credits it. Truth discovery detects copying from many claims shared across many sources; a retrieval agent sees each page once, for one question, so the evidence of copying must come from the page itself. What is new is this per-question provenance count, its guarantees when provenance is incomplete, and its use in the agent's confidence and stopping. In the terms of the ICLR reviewer guidelines, this is new knowledge, empirical and theoretical, with a direct use for practitioners. The reviews agree that the problem matters: Reviewer 1aDv finds it "real and clearly stated", Reviewer 4B32 calls it "well motivated", and Reviewer S9VG finds the distinction between provenance and document similarity "meaningful and practically relevant".

We thank the AC and the three reviewers again for their time and careful reading.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.