- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper propose a probe-driven method to classify the hallucination and deception responses. It utilize a adversarial updating paradigm to generate hallucination-deception pairs and update probe set. Experiments show the effectiveness of proposed method on factual question-answering and science multi-choice questions.
Critical strengths
- It is a new problem to identify hallucination and deception responses.
- The proposed method is clear to understand.
Critical weaknesses
- In practical application, we can simply treat both hallucination and deception as false responses and filter them out. Why do we need to identify hallucination and deception responses separately? Further statements about its significance is needed.
- The proposed method is only tested on limited scopes. Is the proposed method generalized to other domains such as code, math, or open-world questions? More evaluations could be helpful.
- The additional cost can not be ignored. There are 65 additional probe questions needed to sample with the audting model. Analysis on computation costs is needed.
Questions for authors
Please see the weaknesses.