- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper distinguishes hallucinations into confabulation, semantic illusion, and expressed uncertainty, demonstrating differences in detection and mitigation effectiveness across categories.
Critical strengths
The paper provides a systematic empirical evaluation of existing hallucination detection and mitigation methods, revealing meaningful performance differences across distinct error categories.
Critical weaknesses
-
Limited Novelty. The paper primarily repackages well-established phenomena, including prompt sensitivity, knowledge elicitation, and abstention, into a new taxonomy without identifying fundamentally new LLM behaviors.
-
Arbitrary and Budget-Dependent Categorization. Confabulation and semantic illusion are defined using a finite set of prompt variations. Since the prompt space cannot be exhaustively explored, the same question may receive different labels as the evaluation budget increases.
-
Insufficient Evidence for Distinct Mechanisms. The paper does not demonstrate that confabulation and semantic illusion arise from fundamentally different mechanisms. Their differences may simply reflect variations in question difficulty or knowledge accessibility.
-
Predictable Experimental Findings. The observation that prompt-sensitive errors are easier to correct is largely expected from their operational definitions. Similarly, encouraging abstention naturally increases expressed uncertainty without necessarily improving factual accuracy.
-
Unclear Added Value of the Taxonomy. The paper does not establish whether its discrete categories provide additional predictive or diagnostic value beyond continuous measures of prompt sensitivity, answer consistency, and recovery probability.
Questions for authors
NA
3 comments
I have carefully read the authors' response, and I appreciate their efforts to address my concerns. However, I still have the same concerns as before. I think my main concern about the research problem has not been fully addressed. Therefore, I have decided to keep my original score unchanged.
Thank you for your response. Would you let us know which of your concerns remain?
Many thanks, Authors
Thank you for the review. We address below. We tried to be very concise and clear for reviewer's time. We would appreciate it if the reviewer could consider our contribution in their final evaluation.
W1/W4 — Novelty/ unexpected findings
W2 — Budget dependence
W3 — Mechanisms, Q difficulty, and knowledge accessibility
W4 — Our findings differ from what would be expected
W5 — Added predictive and diagnostic value