acceptodds
Withdrawn from ICLR 2027

Aggregate Grounding Scores Hide Citation Turnover

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Grounding benchmarks typically score whether a citation agrees with the benchmark’s evidence annotation under a single presentation. Such marginal scores do not reveal whether the same annotation-correct citations remain correct under content-preserving re-presentation: losses on some questions can be offset by gains on others, leaving aggregate quality nearly unchanged. We formalize this as an identification problem and introduce the Paired Citation Retention Audit (PCRA), which pairs each question’s citation outcome across presentations while holding the reader and available evidence fixed. On 3,330 NExT-GQA questions, two 8B video readers lose 24.2% and 42.0% of their initially correct temporal citations under reordering, corresponding to 3.82% and 5.78% of all question–order pairs, while net hit-rate changes remain near zero. The loss is far larger than under identical-input repeats and minimal perturbation controls, and it persists under exhaustive order coverage and across reader families, packet providers, and model scales. The same pattern appears in text RAG, where six HotpotQA readers lose 4.0–12.6% of their initially correct paragraph citations. Presentation variation also affects reader comparison. On small labelled video sets, averaging grounding quality across multiple presentations improves ranking reliability, whereas paired retention measures whether previously correct grounding decisions are preserved across presentations. The two therefore serve complementary roles: presentation-averaged quality stabilizes reader comparison, whereas PCRA exposes citation turnover that marginal scores cannot identify.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
grounded generation, citation grounding, citation retention, retrieval-augmented generation, multimodal evaluation, presentation sensitivity, evaluation robustness, video question answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bGTvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PCRA to measure whether initially correct citations remain correct after evidence reordering. Experiments on video grounding and text RAG show that citation losses and gains can cancel in aggregate grounding scores. The study also examines repeat variation, presentation averaging, and retrieval-pipeline changes.

Critical strengths

1.The paired analysis reveals instability that aggregate grounding scores cannot measure.

2.The study includes repeat controls and evaluates multiple reader families and modalities.

Critical weaknesses
  1. The measured citation losses may overstate actual grounding failures. Section 7 defines correctness through agreement with benchmark annotations rather than semantic evidence sufficiency. A reordered citation may still support the answer despite falling outside the annotated evidence. The automatic support evaluators show limited agreement, and no human evidence-sufficiency evaluation is provided. It is therefore unclear how much of the reported 24.2% and 42.0% citation loss represents genuinely unsupported answers.

  2. The methodological contribution is limited. The central observation follows from the difference between marginal and paired outcomes: similar aggregate hit rates can conceal changes in which individual instances are correct. The experiments demonstrate this effect across several settings, but do not establish its consequences for actual evidence verification. Without that connection, PCRA remains primarily an annotation-stability diagnostic rather than a demonstrated measure of grounding reliability.

Questions for authors

1.What proportion of lost citations would human annotators judge as still providing sufficient evidence?

2.How sensitive are the reported losses to accepting multiple valid evidence intervals or passages?

3.Do citation changes affect answer correctness or actual evidence verification, rather than only benchmark annotation agreement?

Reviewer_UREqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper points out the problem that the model can perform worse when the evidence is presented differently while preserving the contents. Based on this, the authors propose Paired Citation Retention Audit (PCRA) that calculate the score based on all different combination of order or the given evidence. PCRA reveals the transition map for each grounding question whether re-ordering can change the model performance. The results shows that PCRA can actually capture the instance-level citation continuity and consistency that previous metrics failed to do.

Critical strengths
  1. The paper clearly defines the existing problem and propose new evaluation scheme.

  2. The authors evaluate on various MLLMs and show that model ranking changes with their new metrics.

  3. Comprehensive analysis on aggregate quality, citation continuity, and reader ranking reliability test is presented.

Critical weaknesses

<Severe format problem> The link of all citation, figure, and table are not working in the submitted pdf file, which makes it hard to check the references during the review. This can be the reason for desk-rejection due to inappropriate format usage. Nonetheless, below review and score assessment is written independent from this format issue.

  1. The main limitation is that the paper did not discuss about why presentation sensitivity happens and how to solve this. The fact that reordering of the evidence can change the model prediction (which is argued as first contribution in L123-125) is not something new (as per in related works) and the main contribution of this paper is the application of this finding to video grounding QA, which significantly limits the novelty of this work. L423-429 briefly discuss about mitigation strategies but none of them seems useful. I would recommend the authors to extend L423-429 to find the effective strategies to improve the presentation sensitivity of VLMs.

  2. No consistent patterns found and no explanation for that. Based on PCRA, the paper did various set of analysis with different VLMs. For example, Fig 3A shows that InternVL3VL-8B has higher loss ratio compared to Qwen3VL-8B but the paper simply enumerates these numbers without explaining where this difference come from. Same issue for Fig4. The paper should describe why this change of ranking happens and what does it mean to have the highest or lowest loss of initial hit. L288-294 also found no consistent pattern with model scale and no explanation for that. All of these makes the paper more like technical report not academic writing.

  3. Limited applicabilty in practice. The proposed metric is useful in case where there exists only few questions in the benchmark. In other words, the PCRA is not useful on large evaluation scheme. The citation turnover could be reflected in model ranking with increasing number of questions as it better approximate average model performance and robustness. The paper does not discuss about this scaling benchmark effect on their metric.

Questions for authors
  1. Is there any specific question type that changes the model answer most frequently with reordering?

  2. Fig4 shows that Phi-3.5-Vision remains the lowest ranking and Gemma-4-31B remains the highest ranking with the proposed metrics. Is there any correlation between these two metrics?

  3. What would be the best order of the evidence that model performs the best? Is there any way to exploit this best ordering during inference time?

Reviewer_kW76OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the Paired Citation Retention Audit (PCRA) to measure whether initially correct citations remain correct when the same evidence is reordered. Experiments on video grounding and text RAG show that substantial citation turnover can coexist with nearly unchanged aggregate scores.

Critical strengths
  1. The evaluation question is relevant, and the distinction between aggregate quality and instance-level retention is clearly explained.
  2. The empirical study is extensive, with repeat controls, multiple readers, fresh-sample confirmations, and transparent reporting of limitations and negative results.
Critical weaknesses
  1. Limited conceptual novelty. PCRA adapts established negative-flip analysis to citation evaluation, while the non-identification result follows from standard probability bounds. The application is useful, but a clearer account of the new scientific insight beyond known presentation sensitivity would strengthen the contribution.

  2. Uncertain connection to meaningful evidence loss. The authors correctly distinguish annotation retention from semantic support, but the practical interpretation remains unresolved. Automatic support judges give mixed results, and human evidence-sufficiency labels are unavailable. A focused human evaluation would help distinguish genuine support loss from valid alternative citations.

  3. Limited demonstrated incremental utility. Presentation-averaged quality separates at least as many reader pairs overall, and other forms of averaging provide similar ranking-reliability gains. PCRA is appropriately framed as a diagnostic, but concrete examples of actionable failures missed by standard multi-presentation evaluation would better establish its added value.

  4. Narrow operational evidence. The pipeline-change example demonstrates citation changes, but does not quantify the resulting loss of correctness or semantic support. The small evidence packets and contact-sheet video interface also leave uncertainty about the effect’s magnitude in broader grounding pipelines.

Questions for authors

Please refer to Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.