- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Ruleusion for black-box hallucination detection of short factual answers. It identifies “decoding collapse,” where repeated sampling produces the same incorrect answer, as a limitation of SelfCheckGPT and Semantic Entropy. Ruleusion instead uses hand-written logical transformations to generate related questions with supposedly known expected answers, requiring LLM calls only for answering these questions. The method is evaluated on 5,995 answers from five LLMs across five QA datasets.
Critical strengths
- The paper considers a practically relevant black-box hallucination detection setting.
- The evaluation covers multiple LLMs and QA datasets, providing a reasonably broad experimental setting.
- The motivation of reducing the number of LLM calls is reasonable.
Critical weaknesses
-
The presentation quality is somewhat low. Why are there no references in the first paragraph of the Introduction? Actually, only one paragraph in the Introduction contains citations. After reading through the Introduction, I still cannot clearly understand what the authors are talking about. Are you serious?
-
You mentioned a phenomenon called “decoding collapse,” but there is no further justification for it. A few cases may not be sufficient to demonstrate that this is a meaningful and general phenomenon.
-
In the Introduction, you take SAC3 and MetaQA as examples, but provide almost no detailed description of what they actually do. This makes the discussion quite confusing.
-
What exactly do the authors mean by “test”? This term is repeatedly used, but its meaning is not clearly defined.
-
Is this work mainly vibe-coded? The whole framework feels like a collection of manually designed heuristics, especially given that the transformation rules are simply taken for granted without sufficient justification.
Questions for authors
Please see Weakness and check what you have written in the Introduction.