- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces PCRA to measure whether initially correct citations remain correct after evidence reordering. Experiments on video grounding and text RAG show that citation losses and gains can cancel in aggregate grounding scores. The study also examines repeat variation, presentation averaging, and retrieval-pipeline changes.
Critical strengths
1.The paired analysis reveals instability that aggregate grounding scores cannot measure.
2.The study includes repeat controls and evaluates multiple reader families and modalities.
Critical weaknesses
-
The measured citation losses may overstate actual grounding failures. Section 7 defines correctness through agreement with benchmark annotations rather than semantic evidence sufficiency. A reordered citation may still support the answer despite falling outside the annotated evidence. The automatic support evaluators show limited agreement, and no human evidence-sufficiency evaluation is provided. It is therefore unclear how much of the reported 24.2% and 42.0% citation loss represents genuinely unsupported answers.
-
The methodological contribution is limited. The central observation follows from the difference between marginal and paired outcomes: similar aggregate hit rates can conceal changes in which individual instances are correct. The experiments demonstrate this effect across several settings, but do not establish its consequences for actual evidence verification. Without that connection, PCRA remains primarily an annotation-stability diagnostic rather than a demonstrated measure of grounding reliability.
Questions for authors
1.What proportion of lost citations would human annotators judge as still providing sufficient evidence?
2.How sensitive are the reported losses to accepting multiple valid evidence intervals or passages?
3.Do citation changes affect answer correctness or actual evidence verification, rather than only benchmark annotation agreement?