- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper integrates rule-based checks, bibliographic verification tools, and multiple LLM-based detectors. A learned aggregation model produces a suspicion score for each reference, while class-conditional conformal calibration provides statistical guarantees on false positive and false negative rates. The paper further theoretically analyzes the relationship between detector aggregation and human review rates, and validates the approach on the HallMark and StatRef datasets.
Critical strengths
-
The paper addresses a practically important problem, balancing detection reliability with human review efficiency.
-
The theoretical analysis is relatively comprehensive, providing theoretical support for dual-risk control and the effectiveness of detector aggregation.
-
The experiments are extensive, covering multiple detectors, aggregation strategies, and cross-domain evaluations, with substantial performance improvements on both datasets.
Critical weaknesses
-
The methodological novelty is somewhat limited, as the approach mainly relies on existing detector aggregation and conformal prediction techniques.
-
Practical deployment costs are insufficiently considered. The method relies on multiple LLMs and external verification tools, but lacks a systematic evaluation of inference latency, API costs, and overall efficiency.
-
The deferral rate does not fully reflect the actual human review burden. If references flagged as hallucinated also require manual verification, the reduction in practical review workload may be considerably smaller than the reported results suggest.
Questions for authors
-
Compared with using only a small number of high-performing bibliographic verification tools, do the benefits of integrating multiple LLMs justify the additional computational and API costs?
-
If references flagged as hallucinated are also included in the human review workload, how much efficiency improvement does the proposed method provide in realistic deployment scenarios?
3 comments
Thank you to the authors for the thorough rebuttal and additional experiments. I appreciate the new evaluations on real LLM-generated references, the analysis of actual human review burden, and the measurements of API costs and inference latency. These additions have substantially addressed my practical concerns and strengthened the empirical evaluation.
My main remaining concern is the limited methodological novelty, as the framework largely builds upon existing calibration and aggregation techniques. Moreover, the new results suggest that the additional benefits of integrating multiple LLM judges over a small number of strong verification tools are dataset-dependent and sometimes marginal.
Nevertheless, I believe the paper addresses a practically important problem, and the rebuttal has demonstrated the value and feasibility of the proposed framework. I also believe that research emphasizing practical utility and rigorous empirical validation deserves encouragement, even when the methodological novelty is relatively modest.
Therefore, I will maintain my score of 3. I would also respectfully encourage the other reviewers and the AC to consider the paper's practical significance, and to give it favorable consideration for acceptance.
Thank you for your careful review and for recognizing the practical value of our work.
We agree that our method builds on existing calibration and aggregation work. Our theoretical contribution is the finite-sample deferral analysis (Theorem 3 and Corollary 4), which shows when aggregation improves on a learned single detector.
We agree that the benefit of adding LLM judges depends on the dataset. Since Theorem 1 holds for any detector pool, one can start from the two free verification tools and add judges only when they reduce deferral, as on StatRef (Table 24).
Thank you for the positive assessment. Changes are in blue in the revised PDF.
W1. Novelty
We build on PAC-Wrap's calibration and Sadinle et al.'s oracle. For PAC-Wrap, it calibrates an anomaly score to provide PAC guarantees on FPR and FNR, and reduces ambiguity by loosening error tolerances. We keep the tolerances fixed and contribute a finite-sample analysis of deferral under aggregation.
For Sadinle et al., they provide the oracle and split-conformal class-specific coverage. Our theoretical contribution is the finite-sample deferral bound in Theorem 3, which separates the learned, PAC-calibrated rule's deferral into an information term, aggregation regret, and the unused budget. Corollary 4 gives the exact distribution of that budget.
W2. Runtime, API costs, and overall efficiency
We recorded API costs and running times on StatRef-LLM, a new dataset of 300 LLM-generated references added to evaluate actual LLM citation errors (new Sec. 5.5 and Table 24).
The table below compares the percentage of references flagged+deferred for human review.
Verification tools dominate latency, while LLM judges account for API fees. The tools incur no API fees, but the slower tool takes a median of 51 seconds per reference. The seven judges together cost USD 8.94 per 1,000 references, with median response times ranging from 1 to 20 seconds.
Q1. Do multiple LLMs justify their cost?
Their benefit depends on the dataset. Compared with the two-tool stacker, the full pool reduces 134 human reviews per 1,000 references on StatRef, but provides little additional benefit on HallMark or StatRef-LLM. All seven judges are unnecessary: on StatRef, two tools plus Claude already perform similarly to the full pool. Claude's measured API cost is USD 3.78 per 1,000 references (new Table 24).
W3 / Q2. Human review including flagged+deferred references
We also include the best learned single detector in the comparison of total review burden below.
The benefit of additional LLM judges is data-dependent. On HallMark, all three configurations have similar total review burden. On StatRef, the full pool improves on both alternatives. On StatRef-LLM, aggregation helps, but the two tools already perform similarly to the full pool. These differences likely reflect error types and detector complementarity. HallMark already has a strong single detector. StatRef contains varied metadata errors for which LLM judges may provide complementary evidence. In StatRef-LLM, 85% of hallucinations involve DOI errors, which bibliographic tools can directly check.