- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies hallucination detection in RAG. The authors call the conflict between the given context and the model's own memory the Knowledge-Context Gap. The idea is to compare two internal states of the model: one with the context, and one without it. The paper first shows that removing the context from the prompt and comparing the two sets of SAE features helps detection. But this needs an extra forward pass, and it may change the model's internal computation. So the authors propose RAGap. Instead of removing the context from the prompt, it learns which attention heads read the context, and masks them inside the model. It then extracts SAE features from both states and trains a classifier on them. Experiments on RAGTruth, Dolly, and TofuEval show better results than existing detectors.
Critical strengths
-
The idea of comparing the states with and without context is intuitive. Masking context heads is also cheaper than running the model again without the context.
-
The evaluation covers several benchmarks and models, and RAGap outperforms existing detectors in most settings.
Critical weaknesses
-
It is unclear whether the gap signal actually matters. The ablations show that removing the gap features changes the results by only about 0.01. Also, the heads are masked at only one layer, after earlier layers have already read the context. So the two states may be very similar. This is my main concern.
-
It is unclear where the gains come from. RAGap uses a stronger classifier than baselines such as RAGLens. Without a control that uses the same classifier on features from the normal state only, the gains may come from the classifier rather than from the gap.
Minor: The motivation in the introduction could be stronger. It is not fully clear why modeling the gap is needed beyond existing SAE-based detectors. In addition, some baseline numbers in Table 2 are identical for the 7B and 13B models. Please check them.
Questions for authors
-
If you train the same classifier only on features from the normal state (with context), how does it compare with RAGap?
-
How different are the features before and after masking the context heads? How does masking random heads compare with masking the selected context heads?
I think this paper has room for improvement. Fully addressing the concerns above and presenting a clearer story would make the paper much stronger.
Thank you for your responses, W2 and W3 has been resolved.
Two questions remain: