- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies citation quality in search-augmented LLMs using CITETRACE, a dataset of over 760K citations across ten models. The problem is relevant, and the large dataset and cross-provider analysis are useful. However, the novelty is extremely limited, as Verified Misguidance and its three evaluation dimensions largely repackage existing ideas about citation faithfulness, source reliability, and query-source alignment under new names.
Critical strengths
-
The paper addresses an important problem of citation generation and source attribution in LLMs, particularly how reliably search-augmented LLMs select and use sources to support their answers.
-
The paper introduces a large-scale dataset containing 760k+ citation triples from 11,200 queries across 28 domains and 10 search-augmented LLMs.
Critical weaknesses
-
Verified Misguidance combines already studied problems in citation faithfulness and source reliability under a new name. The paper does not establish sufficient novelty beyond existing work.
-
Several datasets already exist for evaluating citations and source attribution, such as ALCE, LongBench-Cite, SourceBench, REASONS, and others. Although CITETRACE is larger than any of them, the paper does not convincingly explain why existing datasets are insufficient or what makes this dataset meaningfully different. Scale alone can't be considered a meaningful differentiator.
-
The authors do not test whether explicitly asking LLMs to use official or peer-reviewed sources reduces source suitability failures.
-
The authors do not examine whether source suitability failures differ between established factual questions (e.g., vitamin A content in fish oil) and questions involving uncertain predictions (e.g., regional effects of climate change). Official sources may disagree on uncertain topics, but established factual questions generally have verifiable answers. Without separating these cases, the reported failure rates are difficult to interpret.
Questions for authors
-
Can the authors test whether asking LLMs to use only official or peer-reviewed sources reduces source suitability failures?
-
Can the authors report source suitability failure rates separately for established factual questions and questions involving uncertain or evolving evidence?
6 comments
Initial Response (2/3)
(W3/Q1) Source-Selection Guidance and the Rationale for Neutral Evaluation
Neutral Evaluation Measures Baseline Source Selection. CITETRACE aims to measure and characterize structural citation failures in deployed search-augmented LLMs, rather than develop methods to mitigate them. Accordingly, we used a neutral prompt without explicit source-selection guidance to evaluate each system's baseline source-selection behavior. This design allows us to assess the suitability of sources selected by each system without prescribing a particular source-selection policy.
To directly address the reviewer's concern, we conducted an additional experiment with a Source-Guided condition explicitly instructing models to cite official or peer-reviewed sources. We used the same 100 questions across eight models from four providers, drawing the Neutral results from the original evaluation. Both conditions were evaluated using the same citation-processing pipeline and Source Suitability (SS) failure criterion (SS ≤ 2). The two xAI models were excluded because their APIs were no longer available for re-collection.
Table 1. Overall comparison of source-selection guidance
| Setting | SFR ↓ | Official + Research (%) | Evaluable Citations (%) |
|---|---|---|---|
| Full Dataset (Reference) | 27.12% | 26.32% | 59.91% |
| Neutral | 27.72% | 24.64% | 63.25% |
| Source-Guided | 19.80% | 46.34% | 51.49% |
| Δ (Guided − Neutral) | −7.92 pp | +21.70 pp | −11.76 pp |
Table 2. Provider-level SFR comparison
| Provider | Neutral SFR ↓ | Guided SFR ↓ | Δ (pp) |
|---|---|---|---|
| Anthropic | 31.04% | 31.96% | +0.92 |
| 33.25% | 24.43% | −8.82 | |
| OpenAI | 8.45% | 2.24% | −6.21 |
| Perplexity | 29.43% | 6.41% | −23.02 |
Explicit Guidance Does Not Guarantee Consistent Improvements. Explicit source-selection guidance reduced the observed overall SFR, but did not consistently improve source suitability across providers. Despite receiving the same guidance, providers exhibited substantially different responses, with one provider showing an increase in SFR. Moreover, the proportion of evaluable citations decreased under the Source-Guided condition, warranting caution when interpreting the magnitude of the overall SFR reduction. Nevertheless, the observed provider-level differences and remaining suitability failures do not support the explanation that these failures can be consistently resolved across systems through a simple prompt modification.
Why Source Suitability Evaluation Remains Necessary. The effectiveness of explicit source-selection guidance does not eliminate the need to evaluate structural source suitability. Source-selection guidance represents one possible means of improving source-selection quality, whereas CITETRACE provides a framework for systematically measuring that quality across deployed systems. Whether or not explicit guidance is applied, evaluating structural suitability remains necessary to determine whether the selected sources meet domain-specific appropriateness criteria.
Initial Response (3/3)
(W4/Q2) Source Suitability and Evidence Certainty
Source Suitability and Evidence Certainty. Source Suitability (SS) evaluates the structural appropriateness of a cited source's publication type for its content domain, rather than the factual certainty of the user's question. CITETRACE calculates SS based on the combination of a source's content domain (Source Domain) and publication type (Source Type), without assessing whether the answer to a question is well established or whether sources agree with one another. Accordingly, disagreement among official sources on uncertain topics concerns evidence certainty rather than the structural suitability directly measured by SS.
Analysis Across Question Types. Although question-level uncertainty is not an SS evaluation criterion, different question types may lead systems to select different sources and exhibit different failure rates. To examine this possibility, we classified all 11,200 questions using GPT-5 mini into Established Factual (verifiable facts or established knowledge) and Uncertain/Evolving (predictions, contested evidence, or context-dependent judgments). This classification was conducted independently of the original citation evaluations, and we applied the same SS failure criterion (SS ≤ 2) to both groups. We also calculated source-domain-standardized SFRs to account for differences in source-domain composition between the groups.
Table 1. Source Suitability Failure Rates by Question Type
| Question Type | Questions | Evaluable Citations | Observed SFR ↓ | SD-Standardized SFR ↓ |
|---|---|---|---|---|
| Established Factual | 8,124 | 534,244 | 27.49% | 27.29% |
| Uncertain/Evolving | 3,076 | 227,251 | 26.24% | 26.74% |
Interpretation of Results. Separating questions by evidence certainty does not reveal a substantial difference in observed structural source suitability failures. SFR was slightly higher for established factual questions, and the two groups continued to exhibit similar failure rates after standardizing the source-domain distribution. These findings indicate that the reported structural suitability failures are not confined to uncertain or evolving questions, supporting the interpretation of SFR across both question types.
Initial Response (1/3)
We thank Reviewer LT4e for their thoughtful review and constructive questions. We appreciate their recognition of the relevance of the problem and the value of our large-scale dataset and cross-provider analysis. Below, we address concerns about novelty and differentiation from existing benchmarks (W1/W2), the impact of explicit source-selection instructions (W3/Q1), and source suitability across different question types (W4/Q2).
(W1/W2) Novelty and Distinction from Prior Citation Benchmarks
Core Contribution. CITETRACE demonstrates that faithful citations do not necessarily imply appropriate source selection, revealing structural source-selection failures that citation fidelity assessments alone would overlook. Deployed search-augmented LLMs not only generate answers but also select web sources to support them. Even when a citing sentence faithfully represents its source, the source's publication type may be inappropriate for its content domain. CITETRACE explicitly evaluates this structural suitability through an expert-validated domain–source-type matrix, enabling the identification of inappropriate source choices that cannot be determined from citation fidelity alone.
Distinction from Prior Benchmarks. The prior benchmarks considered here do not explicitly evaluate domain-dependent structural source suitability as a distinct quality dimension in deployed search-augmented LLMs. ALCE [1] and LongBench-Cite [2] primarily evaluate citation support against provided evidence, while REASONS [3] focuses on scientific reference retrieval and generation. SourceBench [4] evaluates the relevance and quality of cited webpages, whereas CITETRACE assesses whether a source's publication type is appropriate for its content domain. Unlike individual-source quality assessments, which examine the properties of specific webpages, SS evaluates the structural appropriateness of source selection at the domain–type level. CITETRACE further evaluates this dimension alongside intent–purpose alignment and answer–source fidelity on the same citation instances, enabling systematic analysis across deployed systems.
Empirical Findings. Our findings reveal that structural source-selection failures remain substantial even among citations that pass both intent alignment and citation fidelity evaluations. A considerable proportion of citations fail exclusively on source suitability, demonstrating that faithful citations can still rely on structurally inappropriate sources. Moreover, systems with stronger citation fidelity do not necessarily achieve better structural source suitability, revealing different quality profiles across providers. These results show that citation fidelity alone cannot adequately characterize source-selection quality, and that assessing structural suitability provides additional information about citation failures that would otherwise remain undetected.
We will strengthen the Related Work and contribution statements to clarify these distinctions and the analyses enabled by CITETRACE, and upload the revised manuscript during the rebuttal period.
[1] Gao et al., Enabling Large Language Models to Generate Text with Citations, EMNLP 2023.
[2] Zhang et al., LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-Context QA, Findings of ACL 2025.
[3] Tilwani et al., REASONS: A Benchmark for REtrieval and Automated citationS Of scieNtific Sentences using Public and Proprietary LLMs, arXiv:2405.02228, 2024.
[4] Jin et al., SourceBench: Can AI Answers Reference Quality Web Sources?, arXiv:2602.16942, 2026.