- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces CrossRef-VQA, a benchmark for evaluating cross-modal reference resolution in multimodal large language models under Original, Anonymous, and Disjoint conditions. It analyzes performance degradation and errors when models must resolve anonymized chart labels using external semantic mappings.
Critical strengths
- The paired evaluation design provides a useful comparison of reference resolution with and without external semantic mappings.
- The manually designed question templates and programmatically generated ground truth ensure reproducible evaluation without relying on an LLM judge.
- The detailed error analysis distinguishes different failure types and provides insights across question categories, mapping directions, and chart types.
Critical weaknesses
- The comparison with existing chart question-answering benchmarks is insufficient, and the additional diagnostic value of CrossRef-VQA is not clearly demonstrated.
- Table 2 contains apparent inconsistencies in the O−A values. For example, 78.8−76.7 equals +2.1 percentage points, rather than the reported −0.7, raising concerns about the calculation and the related conclusions.
- The interpretation of Mapping Gap is overstated, as improvements and regressions between conditions A and D may cancel each other out. A near-zero gap alone cannot establish robust mapping or support the claim that the model perfectly bridges the modalities.
Questions for authors
- How does CrossRef-VQA differ from existing benchmarks such as ChartQA and PlotQA, and what additional diagnostic insights does it provide?
- How are the O−A values in Table 2 calculated, and would correcting the inconsistencies affect the conclusions?
- Can the authors report paired outcome statistics for conditions A and D to verify whether a near-zero Mapping Gap reflects robust reference resolution?