acceptodds
Under review as a conference paper at ICLR 2027

Verified Misguidance: Measuring Structural Citation Failures in Search-Augmented LLMs

Abstract

Users of search-augmented LLMs rely on citations as evidence that responses are grounded in real sources, and rarely verify the cited pages themselves. Millions of queries per day now pass through these systems, making citation quality a silent determinant of whether users are informed or misled—yet existing benchmarks address specific facets of citation quality, leaving the joint structure that determines citation trustworthiness unmeasured. We construct CiteTrace, a large-scale dataset that traces the full citation chain from user query through retrieved source to generated answer: 11,200 real-world queries from 28 communities paired with 112,000 responses from ten models across five providers, yielding 761,495 evaluable citation triples. We design a three-dimension evaluation framework that scores each citation on intent–purpose alignment, source suitability, and answer–source fidelity, using expert-validated predefined matrices and a five-level fidelity rubric; the framework applies to any system that produces citation-bearing responses. Applying this framework at scale, we identify a systematic pattern we call Verified Misguidance (VM): models cite real, accessible sources yet fail along one or more dimensions, producing a fidelity–suitability trade-off in which faithful models select inappropriate sources and vice versa. Across our pool, 30.6% of citations distort their sources and 27.1% originate from domain-inappropriate sources; at the response level, up to 96% of users encounter at least one structurally misleading citation. Provider-level differences explain 88–96% of citation-quality variance, suggesting that source selection is governed more by factors beyond individual model capability than by the LLMs themselves. Together, CiteTrace and its evaluation framework provide the first resource for diagnosing structural citation failures in deployed search-augmented systems.

Primary area:
datasets and benchmarks
Keywords:
Citation Evaluation, Search-Augmented Language Language Model, Verified Misguidance, Dataset, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LT4eOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies citation quality in search-augmented LLMs using CITETRACE, a dataset of over 760K citations across ten models. The problem is relevant, and the large dataset and cross-provider analysis are useful. However, the novelty is extremely limited, as Verified Misguidance and its three evaluation dimensions largely repackage existing ideas about citation faithfulness, source reliability, and query-source alignment under new names.

Critical strengths
  • The paper addresses an important problem of citation generation and source attribution in LLMs, particularly how reliably search-augmented LLMs select and use sources to support their answers.

  • The paper introduces a large-scale dataset containing 760k+ citation triples from 11,200 queries across 28 domains and 10 search-augmented LLMs.

Critical weaknesses
  • Verified Misguidance combines already studied problems in citation faithfulness and source reliability under a new name. The paper does not establish sufficient novelty beyond existing work.

  • Several datasets already exist for evaluating citations and source attribution, such as ALCE, LongBench-Cite, SourceBench, REASONS, and others. Although CITETRACE is larger than any of them, the paper does not convincingly explain why existing datasets are insufficient or what makes this dataset meaningfully different. Scale alone can't be considered a meaningful differentiator.

  • The authors do not test whether explicitly asking LLMs to use official or peer-reviewed sources reduces source suitability failures.

  • The authors do not examine whether source suitability failures differ between established factual questions (e.g., vitamin A content in fish oil) and questions involving uncertain predictions (e.g., regional effects of climate change). Official sources may disagree on uncertain topics, but established factual questions generally have verifiable answers. Without separating these cases, the reported failure rates are difficult to interpret.

Questions for authors
  1. Can the authors test whether asking LLMs to use only official or peer-reviewed sources reduces source suitability failures?

  2. Can the authors report source suitability failure rates separately for established factual questions and questions involving uncertain or evolving evidence?

6 comments
AuthorsOpenReview ↗

Initial Response (1/3)

We thank Reviewer LT4e for their thoughtful review and constructive questions. We appreciate their recognition of the relevance of the problem and the value of our large-scale dataset and cross-provider analysis. Below, we address concerns about novelty and differentiation from existing benchmarks (W1/W2), the impact of explicit source-selection instructions (W3/Q1), and source suitability across different question types (W4/Q2).

(W1/W2) Novelty and Distinction from Prior Citation Benchmarks

Core Contribution. CITETRACE demonstrates that faithful citations do not necessarily imply appropriate source selection, revealing structural source-selection failures that citation fidelity assessments alone would overlook. Deployed search-augmented LLMs not only generate answers but also select web sources to support them. Even when a citing sentence faithfully represents its source, the source's publication type may be inappropriate for its content domain. CITETRACE explicitly evaluates this structural suitability through an expert-validated domain–source-type matrix, enabling the identification of inappropriate source choices that cannot be determined from citation fidelity alone.

Distinction from Prior Benchmarks. The prior benchmarks considered here do not explicitly evaluate domain-dependent structural source suitability as a distinct quality dimension in deployed search-augmented LLMs. ALCE [1] and LongBench-Cite [2] primarily evaluate citation support against provided evidence, while REASONS [3] focuses on scientific reference retrieval and generation. SourceBench [4] evaluates the relevance and quality of cited webpages, whereas CITETRACE assesses whether a source's publication type is appropriate for its content domain. Unlike individual-source quality assessments, which examine the properties of specific webpages, SS evaluates the structural appropriateness of source selection at the domain–type level. CITETRACE further evaluates this dimension alongside intent–purpose alignment and answer–source fidelity on the same citation instances, enabling systematic analysis across deployed systems.

Empirical Findings. Our findings reveal that structural source-selection failures remain substantial even among citations that pass both intent alignment and citation fidelity evaluations. A considerable proportion of citations fail exclusively on source suitability, demonstrating that faithful citations can still rely on structurally inappropriate sources. Moreover, systems with stronger citation fidelity do not necessarily achieve better structural source suitability, revealing different quality profiles across providers. These results show that citation fidelity alone cannot adequately characterize source-selection quality, and that assessing structural suitability provides additional information about citation failures that would otherwise remain undetected.

We will strengthen the Related Work and contribution statements to clarify these distinctions and the analyses enabled by CITETRACE, and upload the revised manuscript during the rebuttal period.

[1] Gao et al., Enabling Large Language Models to Generate Text with Citations, EMNLP 2023.

[2] Zhang et al., LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-Context QA, Findings of ACL 2025.

[3] Tilwani et al., REASONS: A Benchmark for REtrieval and Automated citationS Of scieNtific Sentences using Public and Proprietary LLMs, arXiv:2405.02228, 2024.

[4] Jin et al., SourceBench: Can AI Answers Reference Quality Web Sources?, arXiv:2602.16942, 2026.

AuthorsOpenReview ↗

Initial Response (2/3)

(W3/Q1) Source-Selection Guidance and the Rationale for Neutral Evaluation

Neutral Evaluation Measures Baseline Source Selection. CITETRACE aims to measure and characterize structural citation failures in deployed search-augmented LLMs, rather than develop methods to mitigate them. Accordingly, we used a neutral prompt without explicit source-selection guidance to evaluate each system's baseline source-selection behavior. This design allows us to assess the suitability of sources selected by each system without prescribing a particular source-selection policy.

To directly address the reviewer's concern, we conducted an additional experiment with a Source-Guided condition explicitly instructing models to cite official or peer-reviewed sources. We used the same 100 questions across eight models from four providers, drawing the Neutral results from the original evaluation. Both conditions were evaluated using the same citation-processing pipeline and Source Suitability (SS) failure criterion (SS ≤ 2). The two xAI models were excluded because their APIs were no longer available for re-collection.

Table 1. Overall comparison of source-selection guidance

SettingSFR ↓Official + Research (%)Evaluable Citations (%)
Full Dataset (Reference)27.12%26.32%59.91%
Neutral27.72%24.64%63.25%
Source-Guided19.80%46.34%51.49%
Δ (Guided − Neutral)−7.92 pp+21.70 pp−11.76 pp

Table 2. Provider-level SFR comparison

ProviderNeutral SFR ↓Guided SFR ↓Δ (pp)
Anthropic31.04%31.96%+0.92
Google33.25%24.43%−8.82
OpenAI8.45%2.24%−6.21
Perplexity29.43%6.41%−23.02

Explicit Guidance Does Not Guarantee Consistent Improvements. Explicit source-selection guidance reduced the observed overall SFR, but did not consistently improve source suitability across providers. Despite receiving the same guidance, providers exhibited substantially different responses, with one provider showing an increase in SFR. Moreover, the proportion of evaluable citations decreased under the Source-Guided condition, warranting caution when interpreting the magnitude of the overall SFR reduction. Nevertheless, the observed provider-level differences and remaining suitability failures do not support the explanation that these failures can be consistently resolved across systems through a simple prompt modification.

Why Source Suitability Evaluation Remains Necessary. The effectiveness of explicit source-selection guidance does not eliminate the need to evaluate structural source suitability. Source-selection guidance represents one possible means of improving source-selection quality, whereas CITETRACE provides a framework for systematically measuring that quality across deployed systems. Whether or not explicit guidance is applied, evaluating structural suitability remains necessary to determine whether the selected sources meet domain-specific appropriateness criteria.

AuthorsOpenReview ↗

Initial Response (3/3)

(W4/Q2) Source Suitability and Evidence Certainty

Source Suitability and Evidence Certainty. Source Suitability (SS) evaluates the structural appropriateness of a cited source's publication type for its content domain, rather than the factual certainty of the user's question. CITETRACE calculates SS based on the combination of a source's content domain (Source Domain) and publication type (Source Type), without assessing whether the answer to a question is well established or whether sources agree with one another. Accordingly, disagreement among official sources on uncertain topics concerns evidence certainty rather than the structural suitability directly measured by SS.

Analysis Across Question Types. Although question-level uncertainty is not an SS evaluation criterion, different question types may lead systems to select different sources and exhibit different failure rates. To examine this possibility, we classified all 11,200 questions using GPT-5 mini into Established Factual (verifiable facts or established knowledge) and Uncertain/Evolving (predictions, contested evidence, or context-dependent judgments). This classification was conducted independently of the original citation evaluations, and we applied the same SS failure criterion (SS ≤ 2) to both groups. We also calculated source-domain-standardized SFRs to account for differences in source-domain composition between the groups.

Table 1. Source Suitability Failure Rates by Question Type

Question TypeQuestionsEvaluable CitationsObserved SFR ↓SD-Standardized SFR ↓
Established Factual8,124534,24427.49%27.29%
Uncertain/Evolving3,076227,25126.24%26.74%

Interpretation of Results. Separating questions by evidence certainty does not reveal a substantial difference in observed structural source suitability failures. SFR was slightly higher for established factual questions, and the two groups continued to exhibit similar failure rates after standardizing the source-domain distribution. These findings indicate that the reported structural suitability failures are not confined to uncertain or evolving questions, supporting the interpretation of SFR across both question types.

Reviewer_fQ5FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors, through this work, introduce CiteTrace, a dataset of 11,200 Stack Exchange questions, 400 from each of 28 communities. Each question is sent to ten search-augmented LLMs from five providers (OpenAI, Anthropic, Google, xAI, Perplexity). Each citation is scored on three dimensions: Intent: Purpose Alignment, Source Suitability, and Answer–Source Fidelity. The work shows that there is a "fidelity–suitability trade-off" where Anthropic models have low FFR but high SFR, and OpenAI models the reverse and that provider identity explains 88–96% of citation-quality variance with 71–96% of responses contain at least one citation that fails a dimension.

Critical strengths

The main strengths of the work are:

  1. The main strength of the work is the scale and coverage. The work has a total of 112k responses from ten current production systems, gathered in the same two-week window, with the source pages actually crawled rather than taken from a pre-retrieved corpus.

  2. The paper is well presented with clear explanation to the the methodology of the work.

  3. The one think I appreciated was the author's openness about many limitations. They include threshold sensitivity (Table 40), a crawl-failure breakdown by host tier, an audit of unevaluable instances, a check on whether the judge favors its own provider, and a small external replication on Search Arena. The appendix also lists exact model IDs, API settings and prompts.

Critical weaknesses

Given the strengths, there are still a lot of weaknesses that eeds to be addressed:

  1. The comparison across providers is confounded with how the citing sentence is extracted. Table 6 shows two extraction routes. Anthropic is block-based and has the lowest FFR. In Anthropic's API, cited_text is a verbatim excerpt of the source. If that field (or text closely tied to it) is what the judge compares against the page, then high fidelity for Anthropic is close to built in.

A two-sentence window can also pull in text that belongs to an earlier marker or none at all, which would show up as "Fabricated."

To understand this better, the authors need to provide a per-provider human audit of extraction quality and a fidelity re-scored with one shared extraction method, e.g., model-generated text only, located via character offsets, for all providers, before accepting the trade-off as a property of the systems.

  1. The ASF labels, and therefore the provider ranking, look judge-dependent. Table 28 shows Claude Opus 4.7 at κ = 0.463 and Qwen3.5-Plus at κ = 0.429 on ASF, against 0.858 for GPT-4o-mini. The paper reads this as support for choosing GPT-4o-mini. To me it says the fidelity construct is very fragile as the two capable judges disagree with the human consensus about half as much as one would hope.

  2. The human consensus also itself comes from three project-affiliated annotators, and the ASF sample includes only 34 OpenAI citations. Given that, I would like the provider-level FFR ranking recomputed under at least one other judge, and judge agreement reported per provider.

  3. SS is a type-distribution statistic, and the "lower bound" framing is not justified. SS is a fixed lookup on (source domain, source type). SFR is therefore mostly a reweighting of how often each model cites Blog/Social or Company pages; there is no per-source credibility judgment.

  4. "Misguidance" is asserted rather than measured. The paper never checks whether answers are wrong or whether users are misled. There is no correctness label and no user study. An "Amplified" or "Company-source" citation for a correct answer counts the same as one supporting a harmful claim. The construct needs either an answer-correctness signal or a narrower name.

Questions for authors

Please check the weaknesses for questions to address.

5 comments
AuthorsOpenReview ↗

Initial Response (1/3)

We thank Reviewer fQ5F for their careful assessment and constructive suggestions. We appreciate their recognition of the dataset’s scale and coverage, methodological clarity, and our efforts toward reproducibility. Below, we address concerns regarding citation extraction and cross-provider comparability (W1), judge reliability and human validation (W2/W3), the interpretation of source suitability and failure rates (W4), and the scope of Verified Misguidance (W5).

(W1) Citation Extraction and Cross-Provider Comparability

Anthropic's FFR Is Not Based on Source Excerpts. We acknowledge that Table 6 incorrectly describes Anthropic's cited_text field as the evaluated citation unit, and we apologize for the confusion this caused regarding our evaluation procedure. In the actual implementation, we extracted cited sentences from the model-generated response text field, rather than from source excerpts contained in cited_text, and evaluated them using ASF.

Because providers expose citation metadata in different API formats, identifying citation locations required provider-specific processing. However, once citation locations were identified, we applied the same two-sentence extraction window, citation normalization procedure, and ASF scoring criteria across all providers. Thus, ASF evaluates model-generated sentences under a common extraction and scoring protocol, rather than directly comparing Anthropic's source excerpts against their original webpages.

Extraction Audit and Sensitivity Analysis. We conducted additional analyses to assess whether extraction errors could affect cross-provider fidelity comparisons. We randomly sampled 1,000 citations (200 per provider) and checked whether the stored cited sentences and source URLs could be reproduced by reapplying the extraction pipeline to the original API responses. We then manually inspected all 41 cases flagged by these checks, along with 40 additional citations per provider (200 in total), resulting in 241 manually inspected citations. The audit identified three minor formatting discrepancies and one case of omitted preceding context. Restoring the omitted context did not change the ASF judgment, and no incorrect source associations were confirmed among the inspected cases.

We also examined citations whose two-sentence extraction windows contained markers associated with additional URLs. These accounted for 13,016 of 761,495 evaluable citations (1.71%). Excluding them changed the overall FFR only from 30.60% to 30.41%, while preserving the FFR rankings of all five providers and all ten models. These results indicate that multiple URL markers within extraction windows do not materially affect the reported FFR rankings. Together with the extraction audit, this supports the robustness of the provider-level fidelity comparisons underlying the observed fidelity–suitability trade-off.

We will correct the inaccurate description in Table 6 in the revised manuscript, which we will upload during the rebuttal period.

AuthorsOpenReview ↗

Initial Response (2/3)

(W2/W3) Judge Reliability and Robustness of Fidelity Evaluation

Individual Agreement vs. Aggregate Robustness. GPT-4o-mini was selected as the evaluator based on its strong agreement with human annotations for ASF (Cohen's κ = 0.858). However, disagreement between judges on individual ASF labels does not necessarily imply that provider-level FFR comparisons depend on the choice of judge. We therefore distinguish individual judgment agreement from aggregate robustness and conduct an additional cross-judge evaluation to examine whether the major provider-level fidelity patterns persist under a different evaluator.

Cross-Judge Evaluation of Provider-Level FFR. To directly examine whether the provider-level comparisons depend on GPT-4o-mini, we conducted an additional cross-judge evaluation on 2,000 citations (200 per model across ten models and five providers). We compared GPT-4o-mini and Qwen3.5-Plus on the same citations, applying the same ASF failure criterion (ASF ≤ 2), and assessed both individual judgment agreement and changes in provider-level FFRs and rankings. The judges achieved 73.65% agreement (κ = 0.505) on the original five-class ASF labels.

ProviderGPT-4o-mini FFR (Rank)Qwen 3.5 Plus FFR (Rank)Binary AgreementBinary κ
Anthropic10.00% (1)9.75% (1)93.75%0.649
Perplexity30.00% (2)26.50% (2)84.00%0.606
Google32.50% (3)26.75% (3)83.25%0.600
OpenAI41.75% (4)41.00% (5)77.25%0.531
xAI43.75% (5)38.25% (4)77.50%0.536
Overall31.60%28.45%83.15%0.599

Major Provider-Level Fidelity Patterns Persist Across Judges. Our cross-judge evaluation shows that the major provider-level fidelity patterns persist despite disagreement on individual ASF judgments. Anthropic consistently exhibited the lowest FFR under both judges, while OpenAI and xAI remained the two highest-FFR providers despite exchanging positions. These findings demonstrate that the major provider-level fidelity differences are not solely attributable to the choice of GPT-4o-mini as the evaluator.

We will clarify the scope and limitations of the original human validation and incorporate the cross-judge evaluation results into the revised manuscript during the rebuttal period.

AuthorsOpenReview ↗

Initial Response (3/3)

(W4) Interpretation of Source Suitability and SFR

Domain-Dependent Source Suitability. Although SS is derived from domain–type categories, it goes beyond aggregate source-type frequencies by evaluating whether source choices meet expert-validated, domain-specific appropriateness criteria. Source-type frequencies show how often models cite particular types of sources, but do not indicate whether those choices are appropriate for the information domain. CITETRACE uses an expert-validated Source Domain × Source Type matrix that assigns different suitability scores to the same source type across domains. For example, Wiki/Forum sources receive a high suitability score in Code/Data (SS = 5) but a low score in Medical (SS = 1). Accordingly, SFR measures the proportion of citations that fail to meet domain-specific suitability criteria, rather than simply reflecting the frequency of Blog/Social or Company citations.

SS does not assess individual webpage credibility because its objective is to evaluate the structural appropriateness of source selection at the domain–type level. The expert-validated matrix incorporates judgments about the general credibility and authority of source types, but does not distinguish among individual webpages within the same category. While these webpages may differ substantially in factual reliability, assessing such differences falls outside the intended scope of SS. Instead, SS provides a complementary evaluation of whether models select appropriate source types for a given information domain.

Interpretation of SFR. We agree that the observed SFR cannot be interpreted as a conservative lower bound on the failure rate across all citations, given the potential bias introduced by unevaluable sources. The original manuscript used this wording because crawling failures disproportionately affected certain potentially unsuitable source types. However, excluded sources may introduce bias in either direction, and the available data do not establish that the failure rate across the full citation population is necessarily higher. This limitation concerns the generalizability of SFR, rather than the structural suitability failures observed among evaluable citations.

We will clarify the scope of SFR and correct the conservative lower-bound wording in the revised manuscript during the rebuttal period.

(W5) Interpretation and Scope of Verified Misguidance

Meaning of Verified Misguidance. Verified Misguidance (VM) captures potentially misleading citation practices in which real and accessible sources are cited in ways that may be inappropriate or unfaithful to their content. In search-augmented LLMs, citations are presented as evidence supporting generated answers. However, the existence and accessibility of a cited source do not guarantee that the source is appropriate for the information domain or that its content is faithfully represented. Thus, "Verified" refers to the existence and accessibility of the cited source, while "Misguidance" refers to the potential for the citation to misrepresent the appropriateness or evidential support of that source, rather than demonstrated user deception.

Citation Quality vs. Answer Correctness. Answer correctness and citation quality assess distinct aspects of search-augmented LLM responses. Answer correctness evaluates whether the generated information is factually accurate, whereas CITETRACE evaluates how cited sources are selected and used through Intent–Purpose Alignment (IPA), Source Suitability (SS), and Answer–Source Fidelity (ASF). Even a factually correct answer may cite an inappropriate source or make claims that the cited source does not support. These constitute citation-quality failures independently of whether the answer itself is correct. Accordingly, CITETRACE's failure rates quantify violations of citation-quality criteria, not answer inaccuracies or the severity of their potential consequences.

Scope of the VM Construct. Evaluating citation-level failures does not require establishing answer incorrectness or actual user misguidance, because these are distinct evaluation targets. Our study does not directly measure answer correctness or changes in users' judgments, nor does it claim that every identified citation failure results in actual user harm. Rather, VM characterizes conditions under which citations to real and accessible sources may provide misleading signals of evidential appropriateness or support. Answer correctness and user studies would help establish the downstream consequences of these failures, but their absence does not invalidate the identification and evaluation of the citation-quality failures examined by CITETRACE.

We will clarify this distinction, revise wording that could be interpreted as claiming observed user misguidance, and upload the revised manuscript during the rebuttal period.

Reviewer_JjjPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CITETRACE, a large-scale benchmark for evaluating citation quality in search-augmented LLMs. The authors introduce the concept of Verified Misguidance (VM), where LLMs cite real and accessible sources but potentially mislead users through inappropriate source selection or inaccurate representation of source content. To investigate this problem, the paper proposes a three-dimensional evaluation framework covering intent–purpose alignment, source suitability, and answer–source fidelity. The authors collect 11,200 real-world queries from 28 Stack Exchange communities and 112,000 responses from ten LLMs across five providers, resulting in 761,495 evaluable citation triples. The experiments reveal substantial citation-quality problems, including a trade-off between source suitability and citation fidelity. The authors also validate their evaluation framework through human annotation, expert assessment, and additional experiments on Search Arena.

Critical strengths

Important problem and interesting insights. The paper addresses citation reliability from a broader perspective than conventional citation verification.

Comprehensive evaluation and strong reproducibility efforts. The large-scale evaluation across ten models and multiple domains, together with expert validation, external validation, and public dataset release, provides substantial empirical value.

Critical weaknesses

Source suitability is primarily determined by predefined combinations of source domains and publication types. This approach may not adequately capture the actual credibility and relevance of individual sources.

Incomplete source coverage and limitations in interpreting the results. A substantial proportion of cited sources could not be retrieved, potentially introducing systematic bias. Additionally, the observational experiments do not fully establish whether the reported provider-level differences are caused by retrieval infrastructure or other system-level factors.

Questions for authors

I really like the proposed three-dimensional evaluation framework, particularly the distinction between source suitability and citation fidelity. However, I have some concerns about how source suitability is determined. The proposed matrix assigns scores based on source domains and publication types, but sources within the same category can differ substantially in quality. For example, a well-supported technical blog may provide more accurate information than an outdated official document. Although the authors conduct expert validation, could they further investigate whether the predefined scores accurately reflect the quality of individual sources? This would help strengthen the validity of the proposed evaluation framework.

Another concern is the relatively large proportion of citations that could not be evaluated. Section 2.3 reports that approximately 36.8% of citation pairs could not be retrieved, with particularly high failure rates for forums and social media. Since source suitability is one of the central evaluation dimensions, this missing data could affect the reported results. The paper argues that the reported failure rates are conservative lower bounds, but this may not hold uniformly across all domains and models. Could the authors provide additional analysis to quantify the potential impact of missing sources on their main conclusions?

The finding that provider-level differences account for 88–96% of citation-quality variance is interesting. However, I am not fully convinced that these differences can be attributed primarily to retrieval infrastructure. Since providers also differ in their models, retrieval configurations, and generation pipelines, it is difficult to isolate the contribution of each component. Could the authors clarify the interpretation of this result? A controlled experiment using the same retrieval results across different models would help establish whether retrieval infrastructure is indeed the dominant factor.

I would also appreciate clarification regarding the definition of fidelity failures. In Appendix C.3, the proposed rubric distinguishes fabricated, misattributed, contradicted, amplified, and supported citations. However, the Fidelity Failure Rate only considers scores of 1 and 2, excluding contradicted and amplified citations. Since contradicted claims can also be seriously misleading, could the authors explain why these categories are excluded from the main failure metric? Reporting the results under alternative definitions of fidelity failure would help clarify the implications of this design choice.

5 comments
AuthorsOpenReview ↗

Initial Response (1/3)

We thank Reviewer JjjP for their thoughtful evaluation and constructive feedback. We appreciate their recognition of the importance of citation reliability, the broader perspective offered by our framework, and the value of our large-scale empirical evaluation and reproducibility efforts. Below, we address concerns about source suitability and individual-source credibility (W1/Q1), the impact of missing sources on citation failure rates (W2/Q2), the interpretation of provider-level effects (W2/Q3), and the choice of fidelity failure thresholds (Q4).

(W1/Q1) Validity of Source Suitability

Distinct but Complementary Evaluation Scopes. Source Suitability (SS) and individual-source credibility assess distinct but complementary aspects of citation quality: SS evaluates whether a source type is appropriate for an information domain, whereas individual-source credibility assesses the reliability of a specific document. A credible webpage may still be inappropriate for a particular information domain, while a suitable source type does not guarantee the credibility of every webpage within that category. CITETRACE therefore evaluates domain-dependent structural suitability through a predefined source-domain × publication-type matrix, rather than treating SS as a direct measure of individual-document reliability.

Source-Level Credibility Analysis. To examine this distinction, we independently assessed 250 cited webpages using an LLM-based five-point credibility rubric informed by Schwarz and Morris [1]. SS showed only a weak positive association with individual-source credibility (Spearman's ρ = 0.237), and credibility ratings varied within 45 of the 60 source-domain × publication-type categories despite identical SS scores within each category. These findings show that domain–type suitability scores do not fully capture differences in individual-source credibility, underscoring the need to evaluate these complementary aspects separately. Importantly, individual-source credibility alone does not establish whether a source type is appropriate for a given information domain. By assessing this distinct dimension, CITETRACE identifies structurally inappropriate source choices that individual-document credibility assessments alone may overlook.

We will clarify the scope and interpretation of SS, including its distinction from individual-source credibility, in the revised manuscript, which we will upload during the rebuttal period.

[1] Schwarz and Morris, Augmenting Web Pages and Search Results to Support Credibility Assessment, CHI 2011.

AuthorsOpenReview ↗

Initial Response (2/3)

(W2/Q2) Impact of Missing Sources on Citation Failure Rates

Robustness to Missing Sources. Our sensitivity analysis shows that missing sources affect the absolute failure rates but do not eliminate the central fidelity–suitability contrast between Anthropic and OpenAI under the tested common failure-rate assumptions. To assess the impact of missing sources, we conducted a sensitivity analysis accounting for 468,101 uncrawled citations (36.8% of all 1,271,046 extracted citations). The analysis included 761,495 evaluable citations and 468,101 uncrawled citations, excluding 41,450 citations that were successfully crawled but did not pass the final evaluation filters. We retained the original FFR and SFR criteria and recomputed failure rates under assumed failure rates ranging from 0% to 100% for uncrawled citations. The table below reports the results under the representative 50% scenario.

ProviderObserved FFRFFR (50% scenario)Observed SFRSFR (50% scenario)
Anthropic12.90%20.1%31.30%34.9%
Google31.89%42.3%31.37%42.1%
OpenAI43.80%45.6%11.67%22.8%
Perplexity28.24%33.5%30.53%35.2%
xAI41.58%45.5%24.67%36.5%
Overall30.60%37.98%27.12%35.83%

Implications for the Main Findings. Across all tested common missing-source failure-rate assumptions (0–100%), Anthropic consistently exhibits lower FFR but higher SFR than OpenAI. This demonstrates that the central fidelity–suitability contrast persists under the tested assumptions about missing citations, supporting CITETRACE's finding that citation fidelity and source suitability do not necessarily align. The analysis does not, however, establish robustness to provider-specific differences in missing-source failure rates. We therefore do not interpret the observed FFR and SFR as conservative lower bounds for the full citation population.

We will clarify the limitations of the reported failure rates, particularly regarding unevaluable citations, in the revised manuscript, which we will upload during the rebuttal period.

(W2/Q3) Interpretation of Provider-Level Effects

Provider-Level Differences and Causal Interpretation. Our findings establish systematic provider-level differences in citation quality, but do not identify retrieval infrastructure or any other pipeline component as their primary cause. The ANOVA decomposition shows that between-provider differences account for 88.3–96.5% of the variance partitioned between providers and models within providers (Table 36). This quantifies the magnitude of provider-level variation rather than the causal contribution of individual system components.

Scope and Revision. Isolating the effects of retrieval, source selection, response generation, and citation attribution would require controlled experiments, such as fixing retrieved sources across generation models, which are beyond the scope of this study. Nevertheless, the observed provider-level differences remain an important empirical finding, demonstrating that citation quality varies substantially across search-augmented LLM systems regardless of which pipeline components drive these differences.

We will revise the corresponding wording to distinguish observed provider-level differences from causal explanations and upload the updated manuscript during the rebuttal period.

AuthorsOpenReview ↗

Initial Response (3/3)

(Q4) Fidelity Failure Thresholds

Rationale for the Primary FFR Threshold. The ASF ≤ 2 cutoff focuses on Fabricated and Misattributed citations, rather than implying that Contradicted or Amplified citations are less serious. As described in Appendix C.3, the primary FFR targets cases where the cited claim is absent from the source or attributed to an unrelated context. This provides a specific operational definition of fidelity failure, while the five-level ASF rubric separately captures other forms of inaccurate source representation. Contradicted and Amplified citations are therefore not treated as fully faithful, and their inclusion can be examined through broader failure thresholds.

Sensitivity to Broader Failure Definitions. Broadening the FFR threshold increases the measured prevalence of fidelity failures, but does not eliminate the major differences observed across providers. We recomputed FFR on the same 761,495 evaluable citations using ASF ≤ 3 (including Contradicted) and ASF ≤ 4 (also including Amplified).

ProviderASF ≤ 2ASF ≤ 3ASF ≤ 4
Anthropic12.90%13.30%14.95%
Perplexity28.24%29.36%36.31%
Google31.89%33.42%44.50%
xAI41.58%43.22%53.54%
OpenAI43.80%44.34%50.15%
Overall30.60%31.68%38.83%

Under ASF ≤ 3, all five provider rankings and all ten model rankings remain unchanged. Under ASF ≤ 4, some rankings change, but Anthropic still has the lowest FFR, while OpenAI and xAI remain the two highest. These results show that the major provider-level fidelity differences are not dependent on the primary ASF ≤ 2 cutoff. Moreover, the changes in failure rates across thresholds demonstrate the value of CITETRACE's fine-grained ASF rubric, which distinguishes different fidelity failures rather than reducing them to a single binary judgment.

We will clarify the rationale for the primary cutoff and the interpretation of alternative thresholds in the revised manuscript, which we will upload during the rebuttal period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Review and Rebuttal Summary

Dear AC and Reviewers,

We thank the reviewers for their constructive feedback. During rebuttal, we conducted additional analyses, clarified our methodology, and uploaded a revised manuscript.

Reviewer LT4e acknowledged that our additional experiments addressed their experimental concerns, while maintaining concerns about CITETRACE's novelty. We subsequently provided a direct comparison demonstrating its diagnostic value. We also sought to engage in further discussion with Reviewers JjjP and fQ5F and provided detailed responses and manuscript revisions to address their concerns, although we did not receive further feedback from them during the discussion period.

Below, we summarize our main responses, additional experimental evidence, and manuscript revisions.

1. Key Concerns and Responses

  • Novelty and Diagnostic Value (LT4e W1/W2). CITETRACE identifies structural citation failures that existing evaluations may overlook and distinguishes source-selection quality from citation fidelity. It offers two complementary diagnostic benefits:
    • Failure detection: In a direct SourceBench comparison of 1,000 sampled citation triples, 84 of 495 higher-scoring citations (17.0%; SourceBench ≥80/100, our analysis threshold) passed the primary ASF failure criterion (ASF >2) but failed SS (SS ≤2). Favorable individual-source quality does not guarantee structurally appropriate source selection.
    • Performance diagnosis: Perplexity's reasoning model reduced fidelity failures substantially while source suitability remained largely unchanged. CITETRACE identifies persistent source-selection limitations even when citation grounding improves.
  • Complementary Evaluation Scope (JjjP W1/Q1; fQ5F W4/W5). Structural suitability, individual-source credibility, and answer correctness are distinct. SS assesses domain-dependent source-type appropriateness, not individual webpage credibility; an analysis of 250 webpages found only weak correlation (Spearman's ρ=0.237). VM denotes potentially misleading citation practices, not proven answer errors or user deception.

2. Additional Experimental Evidence

EvaluationKey Findings and Implications
Source Guidance (LT4e W3/Q1)SFR 27.72% → 19.80%, with provider-dependent effects and lower evaluable coverage. Guidance helps but does not consistently resolve suitability failures.
Evidence Certainty (LT4e W4/Q2)Established vs. uncertain questions: SFR 27.49% vs. 26.24% (similar after domain standardization). Failures are not limited to uncertain questions.
Extraction Audit (fQ5F W1)241 manually inspected citations showed no confirmed source-association errors. Excluding multi-marker cases changed FFR 30.60% → 30.41%, without changing rankings. Examined extraction issues did not materially alter findings.
Cross-Judge Evaluation (fQ5F W2/W3)Across 2,000 citations, main provider-level patterns persisted under Qwen3.5-Plus. Findings are not solely judge-dependent.
Missing-Source Sensitivity (JjjP W2/Q2)Anthropic–OpenAI contrast persisted under tested common missing-source failure assumptions (0–100%). Robust within these assumptions.
ASF Thresholds (JjjP Q4)Major provider differences persisted for ASF ≤3 and ≤4. Results are not specific to the primary ASF ≤2 threshold.

3. Manuscript Revisions

  • Extraction and Judge Reliability (fQ5F W1–W3): Corrected Anthropic extraction documentation, clarified shared evaluation procedures, and added cross-judge analysis.
  • Provider Effects (JjjP W2/Q3): Clarified that provider-level variation is observational, not evidence of a specific causal mechanism.
  • Evaluation Scope (LT4e W1/W2; JjjP W1/Q1; fQ5F W4/W5): Strengthened comparison with prior benchmarks and clarified SS, individual credibility, and VM.
  • Coverage and Thresholds (JjjP W2/Q2, Q4; fQ5F W4): Clarified crawl-coverage limitations and added alternative ASF-threshold results.

Relevant clarifications and analyses appear in the revised manuscript; rebuttal-only experiments provide further evidence. New text is highlighted in blue and revisions in red.

We appreciate the reviewers' consideration and hope these responses clarify CITETRACE's contributions.

Best regards, Authors of Submission 42337

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.