- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces CompVis-Bench, a benchmark designed to evaluate vision-language models on the structural comprehension of composite visualizations. The benchmark contains two parts. The seed set consists of 121 visualizations selected from VAID and annotated by the authors. The extension set contains 1,600 synthetic visualizations rendered with a randomized renderer. Twelve LVLMs are evaluated on the extension set with metrics at four levels. The main finding is that models recover individual mark groups and their encodings much better than the field-sharing relations between mark groups.
Critical strengths
- The paper addresses a relevant question: whether models can relate the components of a composite visualization through shared data fields.
- The four-level evaluation framework provides a valuable diagnostic lens.
Critical weaknesses
-
Concerns regarding data quality. Many seed images are small, low-resolution crops of figures from published papers. Some contains the original callouts, and some are cut off at the edges. Meanwhile, many extension images have no titles, axis titles, or legends, show randomly generated data, and often combine charts in arbitrary ways. Without these critical information, whether two components share a field often cannot be determined from the image. I also identify several annotations contradict the images. These critical issues must be addressed to make this benchmark usable.
-
Shallow task definition. The benchmark only asks a model to list the components (mark groups) in an image, their encoding channels and repetition, and which components share a data field. It leaves out three aspects that a useful benchmark on composite visualizations should test, and that prior work uses to describe them, such as [*] and [#]. First, it does not test how the components are composed. The annotation that a model must reconstruct records whether a component is repeated, but not whether it is nested in, overlaid on, or placed next to another. Second, it does not test what the shared data mean: a model only has to mark two fields as the same, not say what they represent. Third, it does not test whether a model can use the composition: there is no task that uses the links between components, such as comparing values across them, and the paper places value extraction and question answering outside its scope. A high score on this benchmark would therefore show only a narrow part of what understanding a composite visualization involves.
-
Inconclusive evaluation. The main finding is that models recover local encodings much better than the field-sharing relations between components. However, without a human baseline, it is unclear whether the field-sharing annotations can be recovered from the images at all. The internal-encoding metric only checks which channels each component uses, which largely follows from the chart type, so its high scores show little beyond recognizing the type. None of the local metrics checks which field each channel encodes, so an error such as confusing which axis shows the categories does not lower the local scores but makes any sharing relation involving that field wrong. The gap therefore mixes errors in reading individual components with errors in relating them. Finally, all results come from synthetic images whose annotations were generated by GPT-6-Astra, and the 121 real visualizations are not evaluated, so the gap may reflect properties of the generated data rather than of real composite visualizations.
[*] D. Deng, W. Cui, X. Meng, M. Xu, Y. Liao, H. Zhang, Y. Wu. Revisiting the design patterns of composite visualizations. IEEE TVCG 2023. [#] Z. Wang, Z. Shen, L. Yu, S. Liu. VisPuzzle: Task-Aware Composite Visualization Construction. IEEE VIS 2026.
Questions for authors
-
Can you report results on the 121 seed visualizations? Does the gap between local and field-sharing scores also hold on these real visualizations?
-
Can you report human performance on a random subset of the extension set? This would show whether the field-sharing annotations can be recovered from the images.
3 comments
Thank you for the authors' detailed responses. However, my primary concern is whether the annotated field-sharing relationships can be reliably inferred from the images alone. The authors clarify that the annotations are consistent with the underlying visualization specifications and have been manually inspected. However, this does not establish that these relationships are visually identifiable to independent observers. Without such validation, it remains unclear whether low field-sharing scores reflect limitations in structural understanding or ambiguities in the benchmark itself.
I also share mbYc's concerns regarding the omission of labels and the resulting visual ambiguities. In particular, the concrete examples identified in the subsequent discussion raise further questions about the prevalence of such cases and the reliability of the annotations as evaluation targets. Moreover, although the authors mention exploratory evaluations on real-world seed images, no quantitative results are provided to support the generalizability of the findings.
Since unresolved concerns about visual identifiability and evaluation validity directly affect the benchmark's central contribution and the reliability of its conclusions, I will maintain my original rating of 1.
Clarifying Visual Identifiability and Evaluation Validity
Clarifying Visual Identifiability and Evaluation Validity
We thank the reviewer for the follow-up. We agree that consistency with the underlying specifications and manual inspection do not constitute independent validation of visual identifiability. However, we would like to emphasize that many field-sharing relationships can indeed be inferred from observable visual structures, rather than relying on hidden specifications.
In composite visualizations, shared coordinates, spatial attachment, alignment, and repeated structures provide meaningful evidence of field correspondences. For example, in nested-01-013 and nested-03-016, boxplots and histograms are directly attached to parallel-coordinate axes, making their coordinate relationships visually apparent. In composite-01-013, corresponding facet identifiers and column-wise alignment support the interpretation of shared faceting. These relationships can be inferred from the visual composition itself, even without explicit field labels.
We acknowledge that visual evidence does not always uniquely determine the underlying field identity. However, the existence of alternative interpretations does not necessarily mean that the intended structural relationship cannot be reasonably inferred. Our goal is to evaluate such visually supported structural interpretations, rather than recover arbitrary hidden data provenance. We also recognize that some cases, such as composite-01-005, lack sufficient visual evidence and require correction. We will further clarify our annotation criteria and discuss the resulting limitations.
Regarding the real-world seed images, they were primarily used to derive structural patterns rather than for the main evaluation, partly because of backgrounds, callouts, and other visual artifacts. Our exploratory evaluations suggested similar trends, but we agree that without quantitative results, they cannot independently establish generalizability.
We acknowledge that independent human validation would provide stronger evidence of visual identifiability and will explicitly discuss this limitation. Nevertheless, we maintain that recovering field relationships from visual compositional cues is a meaningful and feasible diagnostic task, and that the identified ambiguities call for more careful annotation and interpretation rather than invalidate the overall evaluation objective.
Clarifications on Dataset Quality, Structural Representation, and Evaluation Validity
Critical Weaknesses
W1. Dataset quality and visual identifiability. The seed and extension sets serve different purposes. The 121 VAID images provide design patterns and structural prototypes, not exhaustive evaluation ground truth. Their quality rarely prevented us from extracting relevant structures; uncertain elements were omitted rather than guessed. Standardized extension rendering avoids source-image artifacts; randomized combinations systematically vary structures rather than optimize design aesthetics.
Our task evaluates anonymous field correspondence, not original field names. Shared axes, spatial arrangements, and repetition can provide cues without textual labels. Some cases may be ambiguous; all three authors inspected and discussed potentially ambiguous sharing relations. We would appreciate specific sample IDs for the reported annotation contradictions so we can examine them.
W2. Task definition and flattening. Our flattened representation is a deliberate methodological contribution, not an omission of composition. The same composite visualization may admit multiple hierarchical descriptions; scoring those descriptions directly can penalize equivalent answers. Flattening provides a canonical output while preserving data-field identities and cross-MG sharing relations, as detailed in the paper.
Repetition is explicitly represented, and its relatively strong results indicate that models can generally produce this part of our representation. Recognizing composition operators, interpreting field semantics, and answering downstream questions are valuable but distinct tasks. We intentionally isolate structural decoding and field correspondence rather than combine every aspect of visualization understanding.
W3. Local encoding, field-sharing metrics, and annotation provenance. The reviewer suggests that local encoding scores largely follow from chart-type recognition and the channel contracts. However, contracts specify available channels, not which are actually used in each mark group. Not all permitted channels are assigned; local encoding evaluation thus requires identifying actual usage, not merely chart type.
We agree that recognizing channels does not establish which fields they encode. Our L3 metrics address this distinction: Field-Sharing Recovery F1 requires correct relations with both endpoints specified by MG, binding type (internal or repetition), and channel under global alignment. MG-Pair Exact Recall requires the complete relation set for each MG pair to match. Confusing channels in a sharing relation therefore reduces L3 scores.
We also report matched-MG variants of both metrics, restricted to successfully matched mark groups. These reduce confounding from MG identification errors, examining correspondence once the relevant MGs are found. They do not eliminate all local channel errors, and we do not interpret the gap as a pure measure of cross-view reasoning difficulty. The metrics instead diagnose different levels of structural reconstruction.
LLM assistance in construction does not mean labels were inferred from rendered images. The extension set uses structured specifications whose field identities are preserved during rendering, and samples underwent manual inspection.
Questions for Authors
Q1. Evaluation on the 121 seed visualizations. We ran exploratory evaluations on the seed images and observed a similar overall trend. We did not report them formally because publication figures contain callouts, extraneous text, inconsistent backgrounds, cropping, and varying resolutions. Such artifacts may confound VLM evaluation even when humans understand the designs. Furthermore, seed annotations were intentionally non-exhaustive: uncertain elements were omitted because the goal was to extract design patterns, not create complete evaluation ground truth. Presenting these scores alongside standardized extension results would be methodologically questionable.
Q2. Human baseline on the extension set. A meaningful human baseline requires experts able to decompose diverse composite designs into mark groups, channels, repetition, and sharing relations. Recruiting experts with this breadth is difficult, and performance may vary with specialization and familiarity with particular designs. A baseline would depend heavily on annotator selection and protocol.
During construction, all three authors inspected potentially ambiguous cases and discussed them to reach consensus. We consider this a reasonable form of qualitative expert validation. It is not equivalent to an independent, statistically representative human baseline, but it provides human scrutiny of the structural annotations rather than relying solely on automatic generation.