- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper discusses the issue of errors in RAG to survive a first round of corrections. It compares the options of rejecting claims that are found to be incorrect, editing them, or revisiting claims that had been ignored in the initial response. A set of experiments is reported on a combination of existing datasets and a variety of models open and closed models. The key finding is that evaluation should not report only the accuracy fn the answer predicted, but also how the coverage. This would give a better assessment of the impact of the edits done on the answers.
Critical strengths
I appreciate the experiments with a wide range of models and datasets.
Critical weaknesses
-
The main weakness of the paper is the lack of novelty: The fact that one should measure both precision (the proportion of what is returned is accurate) and recall (the proportion of what should be returned is correct) in the output of LLMs (and broadly any NLP system) is a well-established practice. There is always a trade-off between returning more information and returning more accurate information in the output, which is why it is advisable to use precision-recall curves and the area under them. The FactScore paper (https://arxiv.org/pdf/2305.14251) makes this clear in its limitations section and it makes the same recommendation, to report both precision and recall
-
While I appreciate the long list of references in the related work section, it would be better to mention fewer and elaborate on how they relate to the work in this paper.
Questions for authors
-
it is unclear why the application context makes a difference to this; what is special about being after the first edit? What is different from just taking the output of an LLM?
-
Could you give an algorithm for the process described in Figure 2? Intuition from a figure is nice for intuition, but an algorithm offer greater precision