acceptodds
Under review as a conference paper at ICLR 2027

HALLMARK: How to Diagnose LLM Citation Verifiers and When to Deploy Them

Abstract

GPTZero found 53 NeurIPS 2025 accepted papers with hallucinated citations (Shmatko et al., 2026; Ansari, 2026), and venues now screen submissions with automated reference checkers. Rule- and large language model (LLM)-based verifiers are emerging, but many are released without a shared evaluation and each flags a different kind of error, so it is unclear which to trust or when to deploy them. We address this with HALLMARK (HALLucination benchMARK): 2,526 BibTeX entries spanning 14 hallucination types, with per-type and diagnostic sub-test labels and further temporal, cross-domain, and real-world evaluation splits. We evaluate zero-shot and tool-augmented LLMs, a DOI-lookup baseline, and our own co-designed rule-based verifier, bibtex-updater. Verifiers can miss hallucinations and wrongly flag correct papers, so every flag still needs human review. The false-positive rate (FPR) determines the review effort and thus limits deployment. The tested models exhibit an order-of-magnitude spread in FPR; agentic lookups buy recall but inflate FPR, and neither smaller models nor prompt optimization reach the frontier’s operating point. A two-stage cascade with bibtex-updater avoids that inflation, providing an upper bound, given the co-design. These rankings, however, depend on what counts as a hallucination: re-scoring the types that describe a real work cited inaccurately changes which verifier ranks first, and venues have not agreed on a definition. We call for a community definition and release per-type labels and a re-scoring protocol so that any agreed definition can be scored against every verifier.

Primary area:
datasets and benchmarks
Keywords:
citation hallucination, hallucination detection, benchmark, large language models, evaluation, tool-augmented agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZMwxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces HALLMARK, a benchmark of 2,526 citations covering 14 types of citation hallucinations, and evaluates different LLMs and database-based verifiers. The paper has a detailed error taxonomy and extensive experiments, with useful findings on false positives, particularly when verifying recent papers or using external tools. However, the motivation for the benchmark is weak, since most of the errors involve bibliographic metadata that can be verified using existing databases. The paper also lacks a strong, independent metadata-matching baseline to justify the use of LLMs for this task. It excludes citation faithfulness, which limits its ability to detect citations that are correct but do not actually support the claims being made.

Critical strengths
  • The benchmark covers 14 types of citation errors and includes six diagnostic checks, allowing a detailed analysis of which errors different verifiers can detect.

  • The paper provides extensive experiments across different LLMs and verification methods. Its analysis of false positives, especially for recent papers and tool-augmented models, offers useful insights into the reliability of citation verifiers.

Critical weaknesses
  • The motivation for the benchmark is very weak, as most of the citation errors involve bibliographic metadata that can be verified through existing databases. The paper does not clearly explain why LLMs are needed for this task. The use of LLMs as verifiers is understandable when checking whether the citation supports the claim, and this setting is explicitly excluded from the benchmark's scope.

  • The paper lacks the main baseline of database lookup and metadata-matching. The DOI-only baseline is too limited to establish whether LLMs offer any advantage over conventional verification methods.

  • Most hallucinated citations are synthetically generated by modifying existing references. This raises concerns about how well the benchmark reflects citation errors encountered in real scientific papers.

  • The best-performing verifier, bibtex-updater, was developed alongside the benchmark taxonomy. This makes it difficult to judge whether its performance and the cascade results would generalize to other datasets.

  • The benchmark excludes citation faithfulness, so it cannot identify citations that refer to real papers but fail to support the claims being made. This limits its usefulness for detecting misleading citations in scientific writing.

Questions for authors
  1. Can the authors evaluate a stronger independent baseline using bibliographic database lookups and metadata matching instead of relying on DOI-only verification?

  2. What specific citation errors in HALLMARK require LLM-based reasoning, and what advantages do LLM verifiers offer over conventional metadata verification?

  3. How well do the results on synthetically generated hallucinations reflect verifier performance on errors found in actual scientific papers?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZMwx (1/2)

We thank the reviewer for their acknowledgement of our work as a fine-grained benchmark with extensive experiments, and for their detailed feedback. We made the following changes to our submission (we detail our responses below):

Summary:

  • We evaluated three independent citation checkers on HALLMARK. RefGuard and reference-audit without its LLM step give the stronger metadata-matching baseline requested.
  • We analyzed recall by error type to show where an LLM step detects that lookup alone misses.
  • We evaluated the same tools on two sets of real citation errors.

Why LLMs are needed (W1/Q2)

The reviewer asks which citation errors require LLM-based reasoning. We agree that lookup alone can catch hallucinated papers (e.g., bibtex-updater, with no LLM, flags all 214 papers that do not exist). In practice, though, lookup tools range from 0.110 to 0.835 recall and from 0.000 to 0.258 FPR on HALLMARK (general response, Table 1). Therefore, whether a database can resolve a citation depends on the tool. Lookup fails in two areas, both of which are addressed by an LLM step.

  • One field wrong. reference-audit's LLM step raises recall on wrong venues (0.09 to 0.89), nonexistent venues (0.08 to 0.81) and preprints cited as published (0.10 to 0.97). Comparing fields by rule cannot distinguish these errors from differences in citation style. Strict matching flags abbreviated venue names and author name variants as errors (lookup tools FPR 0.192 to 0.314 on VerusCite V1; W3/Q3). Without its LLM step, reference-audit notices the venue mismatch on these entries but cannot judge it, and passes most of them as valid (25 of 35 wrong venues).
  • False alarms. A database miss is not proof of a hallucination, because databases have incomplete coverage, as jQVz also notes. The LLM step lowers FPR from 0.070 to 0.013 for reference-audit and from 0.074 to 0.027 for our cascade.

Table 1. Recall on the public test split.

Error typenreference-audit, lookup onlyreference-audit + LLMbibtex-updaterour cascade
Paper does not exist (6 types)2140.100.971.000.85
Wrong venue350.090.890.710.86
Nonexistent venue370.080.810.460.68
Preprint cited as published300.100.970.670.90
Other (5 types)2100.400.910.780.88
All hallucinated5260.2170.9300.8350.854
Valid (FPR)2990.0700.0130.0740.027

Paper does not exist: invented DOI, placeholder authors, a title spliced from several papers, fields merged from two papers, a real DOI with another paper's fields, or a realistic invented paper. Nonexistent venue: the venue does not exist. Other: a future year, missing or swapped authors, a wrong arXiv version, or a real title with small edits. All 14 types are in App. A.1 (Table 2).

Without its LLM step, reference-audit leaves many entries undecided and we count these as not flagged. Our cascade shows the same pattern as reference-audit: the LLM step adds most on single-field types (wrong venue 0.71 to 0.86). It loses some recall on papers that do not exist (1.00 to 0.85) because its LLM step answers "uncertain" on 32 of them, which we count as not flagged.

AuthorsOpenReview ↗

Response to Reviewer ZMwx (2/2)

Baseline of database lookup and metadata matching (W2/Q1)

The reviewer asks for a stronger independent baseline than DOI-only verification. We ran two independent lookup tools without an LLM and two with one (general response, Table 1). None of these tools shares authors with us.

  • Lookup only. RefGuard (Crossref, DataCite, Open Library) reaches recall 0.530 at FPR 0.258. reference-audit without its LLM step (DBLP, Semantic Scholar and others) reaches 0.217 at 0.070. DOI-only reaches 0.110 at 0.000.
  • Lookup plus LLM. reference-audit with its LLM step reaches 0.930 at 0.013. VerusCite, which sends what Crossref cannot confirm to a web-search agent, reaches 0.814 at 0.010.
  • What this shows. Independent lookup tools either miss most errors (reference-audit, recall 0.217) or flag a quarter of valid references (RefGuard, FPR 0.258). With an LLM step, independent pipelines reach recall 0.814 to 0.930 at FPR 0.010 to 0.013, so LLMs improve on conventional verification on both counts (W1/Q2).

Synthetically generated hallucinations (W3/Q3)

The reviewer asks how well results on synthetically generated hallucinations reflect errors in actual scientific papers. Most of our synthetic hallucinations reproduce error types documented in real incidents. Furthermore, to check results on actual errors, the paper already evaluates on authentic ChatGPT citations, and we now add 932 real errors as a new split and evaluate on 2,030 citations from real papers (general response, Table 2).

  • Synthetic entries follow documented error types. Our taxonomy is based on real incidents, as 63JQ also notes. 11 of the 14 types are derived from documented incidents, and they cover about 80% of the hallucinated entries on the public splits. The other 3 are stress-test types. Of the 72 real-world hallucinated citations we analyzed in detail, 79% map to a single type (Sec. 3.1, App. A.2).
  • Results on real errors match HALLMARK. On a new evaluation-only split of 932 hallucinated citations found in published papers (recall only), the lookup tools rank in the same order as on HALLMARK: bibtex-updater 0.924 (0.835 on HALLMARK), RefGuard 0.591 (0.530), reference-audit without LLM 0.113 (0.217), DOI-only 0.089 (0.110). On 341 authentic ChatGPT citations hand-verified by Walters and Wilder, which also include valid entries (App. C.6), the FPR ordering of four LLM verifiers on HALLMARK is preserved (0.029 to 0.860 here), and bibtex-updater reaches recall 0.929 at FPR 0.076 (0.820 at 0.038 on the HALLMARK dev split).
  • 2,030 citations from real papers across fields. On VerusCite V1 (general response, Table 2), lookup tools flag more valid citations outside the ML domain (FPR 0.192 to 0.314), mostly because of differences in citation style (W1/Q2).

Co-design and generalization (W4)

  • Independent tools on HALLMARK. Two independent pipelines with an LLM step reach lower FPR than our cascade (VerusCite 0.010, reference-audit 0.013, vs. 0.027), so the two-stage design works without our tools.
  • Other datasets. bibtex-updater keeps the highest recall on real errors (0.924), but its FPR rises on non-ML citations (W3/Q3 above).
  • Another Stage 1. Swapping our Stage 1 for reference-audit lowers the cascade to 0.751 at FPR 0.171, so the cascade's accuracy depends on the Stage 1 tool.

Citation faithfulness (W5)

We agree that HALLMARK does not test whether a cited paper supports the claim it is cited for. We focus on the earlier step for two reasons. First, a faithfulness checker must retrieve the right paper before it can read it, so an error in this step will affect every later check. Second, this step is not solved, and venues already depend on it. Lookup tools' performance still diverges (W1/Q2), and venues now screen submissions with such tools before human review. They also have no shared definition of a hallucinated citation, yet several have desk-rejected papers for it. Of the eight venues we checked, only two define the term operationally, and their definitions differ. The verifier ranking changes with the definition (App. H.1). However, claim support is an important next step and we now list it as future work.

Thank you for the detailed responses. The additional experiments address most of my experimental concerns. However, I remain concerned about the benchmark's limited scope, as verifying bibliographic metadata alone cannot determine whether a citation supports the claim being made. The practical value of evaluating citation correctness in isolation from citation faithfulness remains insufficiently justified, which continues to affect my overall assessment.

Thank you for engaging with our response, and we are glad the additional experiments address most of your experimental concerns. We agree that citation faithfulness matters. Our Summary of the Discussion above explains why we evaluate citation correctness first, and we list claim support as future work.

Reviewer_63JQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes HALLMARK (HALLucination benchMARK) for citation-metadata verification. It includes 2,526 BibTeX entries spanning 14 error types and three difficulty tiers. It also includes temporal, cross-domain, and real-world evaluation. The paper compares zero-shot LLMs, DOI lookup, tool-augmented LLMs, and a co-designed verifier, bibtex-updater. The false-positive rate (FPR) determines the review effort and thus limits deployment. A two-stage cascade with bibtex-updater achieves the best results.

Critical strengths
  1. The paper provides a detailed hallucination taxonomy with 14 citation hallucination types. The taxonomy is based on real-world incidents. In Appendix H. 1, the paper conducts rescoring for all settings. it shows that changing scoring methods can changes the ranking. The real-world evidence also provides a very useful application. The paper also provides detailed failure mode analysis. The paper has a good model coverage.
  2. The structure of the paper is clear. The paper provides visual cues to help readers understand the results.
Critical weaknesses
  1. The taxonomy, verification sub-tests, and pre-screening were developed alongside bibtex-updater. This might make the results misleading. The paper can benefit from an independently designed first-stage verifier. Most hallucinated entries are synthetic perturbations. C.5.4 shows it makes GPT5.1 19% easier for GPT5.1. Additionally, valid references are chosen from DBLP, which might favor database retrieval systems. To strengthen the idea, the paper can use a frozen verifier against independently collected references, including genuine but poorly indexed publications.
  2. Inconsistency in Appendix D.1: line 2287, it says GPT 5.4 has 22 unresolved. However, in line 2296, GPT 5.4 has 21 unresolved.
Questions for authors
  1. Can the cascade be run with an alternative Stage 1? Since bibtex-updater is Stage 1 of all three cascades and Appendix D implements a deterministic consensus-absence aggregator over CrossRef, OpenAlex and arXiv, can we use that (or DOI-only) as Stage 1 and rerunning Stage 2?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 63JQ

We thank the reviewer for their acknowledgement of our work as a real-world grounded benchmark with thorough analysis, and for their detailed feedback. We made the following changes to our submission (we detail our responses below):

Summary:

  • We ran the cascade with a Stage 1 we did not build, as requested.
  • We evaluated three independent citation checkers on our benchmark.
  • We added 932 real hallucinated citations to the benchmark as a new evaluation-only split, and tested on valid references not sampled from DBLP.
  • We fixed the inconsistency in App. D.1.

Co-design, synthetic perturbations, and DBLP (W1)

The reviewer asks for an independently designed first-stage verifier and independently collected references. We agree that independent tools are needed to address this concern. The general response (concerns 1 and 2) gives the full tables.

  • Independent verifiers. We evaluated three checkers built by others, scored on their own verdicts without our pre-screening layer. Two independent pipelines with an LLM step reach lower FPR than our cascade (VerusCite 0.010, reference-audit 0.013, vs. 0.027), so the benefit of a two-stage evaluation is agnostic to the tools we tested.
  • Independent Stage 1. We ran the cascade with reference-audit as Stage 1 (Q1 below).
  • Real errors. We agree that most hallucinated entries are synthetic perturbations (C.5.4). The paper already evaluates on 341 ChatGPT-generated citations (App. C.6), where bibtex-updater scores higher than on the synthetic dev split (0.929 vs. 0.820). We now add a fifth evaluation-only split to the benchmark, 932 hallucinated citations found in published papers (Hall of Hallucinations). It has no valid ones, so we report recall only. The tools rank in the same order as on our benchmark: bibtex-updater 0.924 recall, RefGuard 0.591, reference-audit without LLM 0.113, DOI-only 0.089.
  • Valid references not sampled from DBLP. This is the test the reviewer asks for. On 2,030 citations from real papers (VerusCite V1, 1,889 valid), bibtex-updater keeps the highest recall (0.752) among the tools, but its FPR rises from 0.074 to 0.314. Our valid entries are sampled from DBLP, so they are written the way the databases record them and exact field matching succeeds, while most V1 citations are non-ML and not in DBLP. On V1, its false flags are mostly venue and author mismatches, the mechanism the paper's cross-domain probe already reported (App. G.2). We add the FPR gap to the limitations.

Cascade with an alternative Stage 1 (Q1)

The reviewer asks whether the cascade can be run with an alternative Stage 1, such as the consensus-absence aggregator or DOI-only.

  • reference-audit as Stage 1. We reran the cascade with reference-audit (no LLM) as Stage 1 and our Stage 2 unchanged. It is an independent lookup tool and a stronger alternative than DOI-only, since it checks every field rather than the DOI alone. The cascade reaches recall 0.751 at FPR 0.171, vs. 0.854 at 0.027 with bibtex-updater. reference-audit forwards 256 of the 299 valid entries to Stage 2 (bibtex-updater: 6), so the cascade's accuracy depends on the Stage 1 tool.
  • Stage 1 decides an entry when it can verify the fields, or rejects it when it finds a mismatch. Everything else goes to Stage 2. DOI-only only checks if the DOI exists and cannot verify the fields. The consensus aggregator only checks whether the title appears in each database. Either way, the run would not test Stage 1, because Stage 2 would check almost every entry and the cascade would reduce to Stage 2 alone.

Minor concerns

Inconsistency in App. D.1 (W2). GPT-5.4 leaves 22 entries unresolved. We corrected the second mention, which said 21.

Thank you for the detailed responses. Most of my concerns have been addressed. I will raise my score to 3.

Thank you for engaging with our response and for raising your score. We appreciate your constructive feedback, which has helped us improve the paper.

Reviewer_jQVzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes HALLMARK, a benchmark to evaluate how reliably automated tools (e.g., LLMs) can detect fabricated or inaccurate citations.The benchmark contains 2,526 BibTeX entries spanning 14 error types, including invented papers and real papers cited with the wrong authors or venue. The authors compare database lookups, standalone LLMs, and a two-stage checker that searches publication records first and sends unresolved cases to an LLM. On the held-out public test set, the two-stage checker detects 99% of citation errors while incorrectly flagging 5.8% of valid references.

Critical strengths
  1. HALLMARK covers 14 error types, from entirely fabricated papers to real papers cited with incorrect authors or venues. The per-type labels make it possible to see which errors a checker catches, and the separate evaluation on hand-verified, ChatGPT-generated citations lets the authors test whether those patterns also appear in real model outputs.

  2. The error analysis identifies a concrete source of false alarms. Bibliographic databases have incomplete coverage, so a valid paper can be missing from one database and present in another. Holding the search results fixed, the authors show that flagging a citation whenever any database misses it produces a 72.9% false-positive rate. Requiring all three databases to miss it reduces that rate to 4.9%, but recall also falls from 83.5% to 25.1%. This experiment shows how the checker’s decision rule affects both false alarms and missed errors.

Critical weaknesses
  1. The contributions of the benchmark is limited. The authors obtain valid references from DBLP and generate many of the errors by modifying fields such as the title, authors, or venue. This is a reasonable way to build a controlled evaluation, but the construction itself has small novelty or value. It does not contain a unique idea or some sort of extraordinary effort. Several error types also appear relatively easy for current models. On the development set, zero-shot Sonnet 4.6 detects 91.6% of errors.

  2. The paper is difficult to read. The frequent use of colons, semicolons, em dashes, and shifts in typography makes already dense passages harder to follow. Notably, the first eight pages do not include and example of the dataset. The reader has to turn to Appendix Table 2 to see all 14 definitions together. A worked example early in the paper would make the dataset and the subsequent results much easier to understand.

  3. The two-stage checker is a sensible engineering approach, rather than a research contribution. Running a rule-based checker first and forwarding uncertain cases to an LLM is a simple design not much novelty.

  4. Some findings are easily predictable. Flagging a reference whenever one database fails to find it will produce false alarms when databases have incomplete coverage. Zero-shot LLMs would obviously fail to do such knowledge recall tasks, especially if some papers may come beyond their knowledge cutoff.

Questions for authors

see weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jQVz (1/2)

We thank the reviewer for acknowledging our work as a per-type benchmark with a concrete analysis of false alarms, and for the detailed feedback. We made the following changes to our submission (we detail our responses below):

Summary:

  • We applied the re-scoring protocol, one of our stated contributions, to current venue policies. App. H.1 now scores every verifier under the ICML 2026 and EMNLP 2026 definitions, which rank 13 of 19 configurations differently.
  • We moved a worked example of a benchmark entry into the main text and cut the punctuation and font changes the reviewer flagged.

Limited contribution; error types relatively easy (W1)

The reviewer finds the construction of small novelty and several error types relatively easy. We agree that perturbing DBLP fields is not a new method. Which fields to change, and how, follows a taxonomy derived from documented incidents. Of 72 real hallucinated citations we analyzed in detail, 79% map to a single one of our 14 types (Sec. 3.1). The novelty lies in what this makes possible (below).

Novelty.

  • The first benchmark that can be rescored under competing definitions. Venues now reject papers for hallucinated citations, but most do not define the term, and the two that do (ICML and EMNLP 2026) disagree, for example on swapped authors. Scored under each definition, the two venues rank 13 of 19 verifier configurations differently (App. H.1). To our knowledge, HALLMARK is the first citation benchmark to map current venue policies onto its error types, show this, and release a re-scoring protocol, built on the per-type labels the reviewer lists as a strength.

    Zero-shot LLMs whose rank differs most between the two venues (rank among all 19 configurations in parentheses):

    Zero-shot LLMMCC as shippedMCC under ICML 2026MCC under EMNLP 2026
    Gemini 2.5 Pro0.417 (12)0.393 (2)0.338 (9)
    GPT-5.40.537 (9)0.378 (3)0.332 (10)
    Llama 4 Maverick0.433 (11)0.352 (8)0.205 (15)
    Gemini 2.5 Flash0.385 (14)0.336 (9)0.207 (14)
  • Diagnostics that show why a verifier fails. On the verifier side, holding the database lookups fixed isolates the decision rule as the source of false alarms. Flagging on any database miss gives FPR 0.729, and requiring all three to miss cuts FPR to 0.049 but recall to 0.251 (App. D). On the entry side, six sub-test labels per entry record which fields are wrong, so detection can be split by how many fields an entry breaks. Real-world hallucinations can break more than one field (App. C.5.3).

Difficulty. High recall does not make the task easy.

  • False alarms. Zero-shot Sonnet 4.6 reaches recall 0.910 on dev but FPR 0.151. If 2% of citations are hallucinated, only about 11% of its flags are real errors.
  • A perfect false-alarm record is not enough. DOI-only raises no false alarm on either public split, 0 of 299 on test and 0 of 501 on dev, so every entry it flags is a real error. It still misses almost 9 in 10 hallucinations, at recall 0.110. No verifier we score is strong on both counts.
  • Hard types remain. bibtex-updater detects 0.12 of partial author lists, and no calibrated standalone verifier exceeds 0.25 on real swapped authors (App. C.6). Even Tier 1 is not solved. Gemini 2.5 Pro passes 14 of 29 fabricated DOIs as valid, while DeepSeek V3 and Qwen3, which catch nearly all Tier 1 entries, flag over 60% of valid references.
AuthorsOpenReview ↗

Response to Reviewer jQVz (2/2)

Predictable findings (W4)

The reviewer finds some findings easily predictable. We agree that the direction of both effects can be expected. What the benchmark adds is information about the size of the effects and the form that failure takes for LLMs.

  • Zero-shot LLMs past their cutoff. One would expect LLMs to miss hallucinated papers they have never seen. They do not. On 2024 to 2025 papers, recall holds or rises (e.g., GPT-5.1 0.827 to 0.958). They fail on valid papers instead. Earlier-cutoff models flag unfamiliar valid entries as hallucinated, and GPT-5.1's FPR rises from 0.413 to 0.759, while the two latest-cutoff models keep FPR at 0.120 and 0.073 (Sec. 4.4, App. F.1).
  • Database coverage. We agree that flagging on a single database miss is expected to raise false alarms. What the benchmark adds is the size of the trade-off (W1 above, App. D). The reviewer's second strength credits this same experiment.

Two-stage checker as an engineering approach (W3)

The reviewer views the two-stage checker as a sensible engineering approach rather than a research contribution. We agree that the design is simple. We use the benchmark to examine each stage, changing one at a time to find which part produces the cascade's accuracy.

  • Stage 1 decides the result. With reference-audit as Stage 1 instead of bibtex-updater, recall drops from 0.854 to 0.751 and FPR rises from 0.027 to 0.171 (general response, Table 1). reference-audit forwards 256 of the 299 valid entries to Stage 2, bibtex-updater 6.
  • The LLM step also lowers false alarms in an independent pipeline. reference-audit's own LLM step lowers its FPR from 0.070 to 0.013.
  • The Stage 2 model changes FPR little. With the same Stage 1 (bibtex-updater forwards the same 211 entries it cannot decide to Stage 2), GPT-6 Luna, Sonnet 4.6 and GPT-5.4 as Stage 2 reach FPR 0.027 to 0.033 on the public test split. Recall differs more (0.854 to 0.966), mainly in how often the model answers "uncertain".

None of this follows from the design itself, and the benchmark allows future tools to be evaluated the same way. Sec. 4.5 now states that the cascade's results depend on Stage 1.

Difficult to read, no early example (W2)

  • Worked example. Before any results, Sec. 3.1 now shows a near_miss_title entry (Attention is All you Require against the real ...Need) and which sub-tests it passes and fails.
  • All 14 types in the main text. Sec. 3.1 now defines every type by tier, so the reader no longer needs App. Table 2.
  • Punctuation. We replaced colons, semicolons and em dashes in dense passages with full sentences.
  • Typography. We removed the link boxes and set tool names and verdict labels in plain text, so only error-type and split names change font.

Thanks for the reply. However, I still find the paper limited in novelty and contribution. Therefore, I keep my score.

Thank you for your reply. Our Summary of the Discussion above outlines how we see the paper’s contributions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to the AC and all reviewers

We thank the reviewers for recognizing our work: the taxonomy based on real-world incidents (63JQ, ZMwx, jQVz), the extensive experiments across LLMs and verification methods (ZMwx, 63JQ), the false-positive analysis (ZMwx), and the error analysis that identifies a concrete source of false alarms (jQVz). Their detailed feedback improved our submission, and the table below summarizes how we addressed the concerns.

Improving our submission

ConcernWhat we didOutcome
1. Co-design of our tool and cascade with the benchmark (63JQ W1/Q1, ZMwx W4)Ran three checkers built by others and swapped our Stage 1 for reference-audit.Two independent pipelines with an LLM step reach a lower FPR than our cascade (Table 1), so the two-stage design works without our tools. Swapping Stage 1 lowered recall and raised FPR, so the cascade's accuracy depends on the Stage 1 tool.
2. Synthetic vs. real errors (ZMwx W3/Q3, 63JQ W1)Added 932 real hallucinated citations as an evaluation-only split, and ran our tools on an external benchmark of 2,030 citations from real papers.The types derive from real incidents (ZMwx W3/Q3; Sec. 3.1). On real errors, our tool scores higher than on the synthetic split (Tables 1-2; 63JQ W1). However, all lookup tools except DOI-only overflag non-ML citations (App. G.2; limitation added).
3. No baseline stronger than DOI-only (ZMwx W2/Q1)Ran reference-audit with its LLM off and RefGuard.Independent lookup tools beat DOI-only on recall but at a much higher FPR (Table 1).
4. Why LLMs, if databases can check metadata (ZMwx W1/Q2)Compared each pipeline with and without its LLM step, by error type.As jQVz notes, databases have incomplete coverage, so database results alone cannot catch most errors without also flagging many valid citations (App. D). Adding an LLM step decreases FPR and catches errors in a single field (e.g., a wrong venue).

How independent reference-audit is. It shares no authors with us, but its developer debugged it on our public dev split. We report v0.23, the first version scored on the public test split. Our Stage 2 prompt lists the 14 error types, and theirs do not.

Table 1. Independent pipelines with an LLM step reach a lower FPR than our cascade

GroupVerifierLLM stepRecallFPRPPV at 2%
No LLMDOI-onlynone0.1100.0001.00 (0 FP)
bibtex-updater v1.2.0 (ours)none0.8350.0740.188
reference-audit v0.23none0.2170.0700.059
RefGuardnone0.5300.2580.040
Built-in LLM, independentVerusCiteweb-search agent0.8140.0100.623
reference-audit + its LLM (GPT-6 Luna)checks DB matches0.9300.0130.586
Two-stage, our Stage 2bibtex-updater (ours) → GPT-6 Luna5-call agent0.8540.0270.394
reference-audit → GPT-6 Luna (Stage 1 swap)5-call agent0.7510.1710.082

All verifiers are scored on the 825 entries of the public test split (526 hallucinated, 299 valid), using our corrected labels (v1.2.5). Entries a tool cannot decide count as not flagged, so every tool is scored on the same entries. VerusCite was run on Nov 8, 2026, and anything but verified counts as a flag, as its developer counts it. PPV at 2% is the share of flags that are real errors if 2% of citations are hallucinated.

Table 2. Recall on real non-ML citations is close to the synthetic split, but FPR is higher

VerifierReal errors: RecallV1: RecallV1: FPR
DOI-only0.0890.0640.003
RefGuard0.5910.3830.286
reference-audit v0.23 (no LLM)0.1130.1350.192
bibtex-updater0.9240.7520.314

Real errors are 932 hallucinated citations found in published papers, collected by VerusCite (Hall of Hallucinations) and added as an evaluation-only split with no valid entries. In a manual check of 100, 98 were hallucinated. V1 is VerusCite's own public benchmark: 2,030 citations from 36 real papers (141 hallucinated, 1,889 valid). We do not score VerusCite here, because both sets were labeled with its help.

AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers and the AC for the discussion.

63JQ and ZMwx state that the revision addresses most of their experimental concerns, on co-design, real-world errors and baselines. 63JQ raised the rating to weak acceptance.

We see the paper's main contributions as the benchmark, the failure-mode analysis and the re-scoring protocol, and the revision shows that they hold without our own tools. We thank all three reviewers for noting these among the strengths. The revision also applies the re-scoring protocol to current venue policies. To check that the findings do not rely on our own tools or on synthetic errors, we added three verifiers built by others and real hallucinated citations from published papers, where the ranking holds. On false alarms, two independent pipelines rank above our own cascade. The cascade is not among these contributions, and the one finding we keep from it is that its results depend on the Stage 1 tool. The final version will present bibtex-updater as one of the baselines and move the independent results into the main text.

One remaining point concerns scope, and HALLMARK targets the step that every faithfulness check depends on. We agree that whether a citation supports its claim matters. This paper focuses on an earlier question, which verifiers venues and authors can trust when they check citations for hallucinations. Citation correctness limits how reliable any faithfulness check can be, so we evaluate it first. This step is also not solved. Most verifiers either miss many hallucinated citations or flag many valid ones, and every flag needs manual review. What counts as a hallucinated citation is also still being defined, and current venue policies differ. Checking claim support is an important next step, and we list it as future work.

The revised PDF, uploaded with this summary, includes all changes described in our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.