- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
HalluciBench gives models a paper title and asks them to produce one BibTeX entry or refuse. The dataset contains 1,000 real ML papers and 200 synthetic papers, with five prompt variants for each. Some variants also provide an author hint, the venue and year, or the first author and year. The paper tests 33 locally deployable models without search, RAG, or access to the papers themselves. It reports BibTeX parsing, metadata accuracy, and refusal scores.
Critical strengths
-
The paper uses five prompts for the same paper, so readers can compare what happens when some author, venue, or year information is added.
-
It reports format and individual metadata scores separately, which gives more information than a single overall accuracy score.
Critical weaknesses
-
I find the practical relevance and advance over prior work insufficient.
In actual citation work, a user who knows the paper title can consult the paper or look up its bibliographic record. This study removes access to those sources and asks local models to reconstruct the reference from memory. The authors explicitly justify this restriction as a way to isolate parametric recall and describe the benchmark as complementary to tool-assisted citation systems. I remain unconvinced that the insights from this restricted task establish sufficient practical value to warrant a new benchmark.
The code link has expired, and I find the dataset contribution limited. The code link in the abstract currently returns HTTP 410 with
repository_expired, so I cannot obtain the code or dataset through it. Existing resources address related but different needs: S2ORC is a scholarly corpus, ArXivBench evaluates research-paper and arXiv-link generation, and ArxEval includes jumbled- and mixed-title tasks. HalluciBench's exact-title-to-BibTeX task and matched no-answer controls are distinct. However, I find the evidence that this combination yields sufficiently new insights beyond existing citation-fabrication and bibliographic-recall studies unconvincing.The evaluation also leaves out current frontier proprietary models. It does not show how those commercial systems perform, so its value as a reference for current citation tools is limited.
Several central questions have already been studied. Agrawal et al. examined fabricated references and author consistency. Niimi examined the relationship between citation count and recall, and differences in how bibliographic fields are remembered. HalluciBench cites both and adds a fixed title-anchored pool, matched refusal controls, prompt variants, and broader model comparisons. These additions provide a distinct evaluation setup, but I find the resulting analysis of failure mechanisms too limited to constitute a substantial advance over the earlier evidence on citation fabrication and uneven bibliographic recall.
-
The analysis does not sufficiently disentangle failure causes and lacks answer-level joint error statistics.
Much of this task depends on memory. A model may never have seen the paper. Even if it has seen the main text, it may not have seen the complete authors, final venue, or DOI. Even if it has seen those details, it may not remember them all or be able to put them into BibTeX. The two-year rule does not solve this: publication at least two years before a model's public release does not establish training-corpus inclusion. The authors acknowledge this, but different training coverage can still affect the model comparison. This is a problem with several possible causes, but the authors do not use enough experiments to separate them. The reported results still mix these causes and the resulting factual errors.
The analysis of memory differences stops too early. Well-known papers may be remembered better, and different kinds of bibliographic information may be remembered differently. The paper groups results by citation count and year, but does not further examine when models are more likely to make up the remaining details, depending on which details they can already get right.
The existing metrics need answer-level error breakdowns. The paper separately reports refusal, parseability, and field-level precision/recall/F1. Its worked example gives full author precision and low recall to “Vaswani et al.”, acknowledging correct but incomplete authorship. However, these summaries do not show how omissions and false additions coexist within individual answers. Non-parseable, non-refusal attempts also receive zero field credit; this measures output usability and does not establish that their factual content is false. Complete correct answers, correct but incomplete answers, mixtures of correct and false details, and answers with only false new details should be counted separately. This breakdown should identify answers that only copy the prompt and retain the paper's separate refusal and format-failure strata.
Additional comparisons would clarify practical relevance and failure causes. The closed-book restriction is deliberate. To assess how far the findings extend to practical citation systems, I would like the same papers tested in four settings: title only; the full paper in context; internet search enabled; and RAG enabled. Under the same information, the model should also be asked to list the metadata directly and to generate BibTeX. This would help tell whether the problem is obtaining the information or writing the format. In the paper-context condition, only details actually present in the paper should count as supplied information. The existing partial-metadata prompts do not replace these comparisons.
The synthetic papers raise a separate issue about the refusal labels. The authors did manual title checks and Semantic Scholar searches. However, more evidence is needed that the controls do not match real papers and resemble realistic citation mistakes. If a model returns a real paper with a similar title, that is a wrong match and should be distinguished from inventing a nonexistent paper. The paper also does not show whether the synthetic titles are easy to identify through style. These questions affect what the refusal scores mean.
Questions for authors
-
Beyond the existing field-level metrics, among parseable citations that add metadata beyond the prompt, what proportions are completely correct, correct but incomplete, mixed correct/incorrect, or contain only incorrect new details? Please distinguish answers adding no new information and retain the existing separate refusal and format-failure strata. Does the proportion of false remaining details change depending on which details were correctly recovered?
-
Can you provide the full-paper, internet-search, and RAG comparisons described above, at least on a representative subset, and compare direct metadata output with BibTeX?
-
Were model outputs pointing to real papers with similar titles checked, and how were possible collisions and stylistic cues in the synthetic controls assessed?