acceptodds
Under review as a conference paper at ICLR 2027

CompVis-Bench: Benchmarking Structural Understanding of Composite Visualizations

Abstract

Many real-world visualizations combine charts, maps, and glyphs whose compo- nents are connected through shared data fields. Understanding these composite visualizations requires recovering not only each component’s visual encoding, but also the data relationships that connect the components. We introduce COMPVIS- BENCH, a reconstruction-based benchmark for evaluating this structural under standing in vision-language models. Its normalized mark-group representation preserves data semantics, encoding structure, and cross-group field correspon dences while abstracting away presentation details and alternative composition hierarchies. We build an expert-annotated seed set and extend it through con trolled structural modifications, randomized rendering, and manual verification, yielding 1,600 verified visualizations. Across 12 large vision-language models, local mark groups and encodings are recovered substantially more reliably than the field correspondences that connect them. These results expose a central lim itation of current models: recognizing the parts of a complex visualization does not imply understanding how those parts jointly represent data.

Primary area:
datasets and benchmarks
Keywords:
Composite Visualization Understanding,Vision-Language Models,Structural Semantics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jWnFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces CompVis-Bench, a benchmark designed to evaluate vision-language models on the structural comprehension of composite visualizations. The benchmark contains two parts. The seed set consists of 121 visualizations selected from VAID and annotated by the authors. The extension set contains 1,600 synthetic visualizations rendered with a randomized renderer. Twelve LVLMs are evaluated on the extension set with metrics at four levels. The main finding is that models recover individual mark groups and their encodings much better than the field-sharing relations between mark groups.

Critical strengths
  • The paper addresses a relevant question: whether models can relate the components of a composite visualization through shared data fields.
  • The four-level evaluation framework provides a valuable diagnostic lens.
Critical weaknesses
  • Concerns regarding data quality. Many seed images are small, low-resolution crops of figures from published papers. Some contains the original callouts, and some are cut off at the edges. Meanwhile, many extension images have no titles, axis titles, or legends, show randomly generated data, and often combine charts in arbitrary ways. Without these critical information, whether two components share a field often cannot be determined from the image. I also identify several annotations contradict the images. These critical issues must be addressed to make this benchmark usable.

  • Shallow task definition. The benchmark only asks a model to list the components (mark groups) in an image, their encoding channels and repetition, and which components share a data field. It leaves out three aspects that a useful benchmark on composite visualizations should test, and that prior work uses to describe them, such as [*] and [#]. First, it does not test how the components are composed. The annotation that a model must reconstruct records whether a component is repeated, but not whether it is nested in, overlaid on, or placed next to another. Second, it does not test what the shared data mean: a model only has to mark two fields as the same, not say what they represent. Third, it does not test whether a model can use the composition: there is no task that uses the links between components, such as comparing values across them, and the paper places value extraction and question answering outside its scope. A high score on this benchmark would therefore show only a narrow part of what understanding a composite visualization involves.

  • Inconclusive evaluation. The main finding is that models recover local encodings much better than the field-sharing relations between components. However, without a human baseline, it is unclear whether the field-sharing annotations can be recovered from the images at all. The internal-encoding metric only checks which channels each component uses, which largely follows from the chart type, so its high scores show little beyond recognizing the type. None of the local metrics checks which field each channel encodes, so an error such as confusing which axis shows the categories does not lower the local scores but makes any sharing relation involving that field wrong. The gap therefore mixes errors in reading individual components with errors in relating them. Finally, all results come from synthetic images whose annotations were generated by GPT-6-Astra, and the 121 real visualizations are not evaluated, so the gap may reflect properties of the generated data rather than of real composite visualizations.

[*] D. Deng, W. Cui, X. Meng, M. Xu, Y. Liao, H. Zhang, Y. Wu. Revisiting the design patterns of composite visualizations. IEEE TVCG 2023. [#] Z. Wang, Z. Shen, L. Yu, S. Liu. VisPuzzle: Task-Aware Composite Visualization Construction. IEEE VIS 2026.

Questions for authors
  • Can you report results on the 121 seed visualizations? Does the gap between local and field-sharing scores also hold on these real visualizations?

  • Can you report human performance on a random subset of the extension set? This would show whether the field-sharing annotations can be recovered from the images.

3 comments
AuthorsOpenReview ↗

Clarifications on Dataset Quality, Structural Representation, and Evaluation Validity

Critical Weaknesses

W1. Dataset quality and visual identifiability. The seed and extension sets serve different purposes. The 121 VAID images provide design patterns and structural prototypes, not exhaustive evaluation ground truth. Their quality rarely prevented us from extracting relevant structures; uncertain elements were omitted rather than guessed. Standardized extension rendering avoids source-image artifacts; randomized combinations systematically vary structures rather than optimize design aesthetics.

Our task evaluates anonymous field correspondence, not original field names. Shared axes, spatial arrangements, and repetition can provide cues without textual labels. Some cases may be ambiguous; all three authors inspected and discussed potentially ambiguous sharing relations. We would appreciate specific sample IDs for the reported annotation contradictions so we can examine them.

W2. Task definition and flattening. Our flattened representation is a deliberate methodological contribution, not an omission of composition. The same composite visualization may admit multiple hierarchical descriptions; scoring those descriptions directly can penalize equivalent answers. Flattening provides a canonical output while preserving data-field identities and cross-MG sharing relations, as detailed in the paper.

Repetition is explicitly represented, and its relatively strong results indicate that models can generally produce this part of our representation. Recognizing composition operators, interpreting field semantics, and answering downstream questions are valuable but distinct tasks. We intentionally isolate structural decoding and field correspondence rather than combine every aspect of visualization understanding.

W3. Local encoding, field-sharing metrics, and annotation provenance. The reviewer suggests that local encoding scores largely follow from chart-type recognition and the channel contracts. However, contracts specify available channels, not which are actually used in each mark group. Not all permitted channels are assigned; local encoding evaluation thus requires identifying actual usage, not merely chart type.

We agree that recognizing channels does not establish which fields they encode. Our L3 metrics address this distinction: Field-Sharing Recovery F1 requires correct relations with both endpoints specified by MG, binding type (internal or repetition), and channel under global alignment. MG-Pair Exact Recall requires the complete relation set for each MG pair to match. Confusing channels in a sharing relation therefore reduces L3 scores.

We also report matched-MG variants of both metrics, restricted to successfully matched mark groups. These reduce confounding from MG identification errors, examining correspondence once the relevant MGs are found. They do not eliminate all local channel errors, and we do not interpret the gap as a pure measure of cross-view reasoning difficulty. The metrics instead diagnose different levels of structural reconstruction.

LLM assistance in construction does not mean labels were inferred from rendered images. The extension set uses structured specifications whose field identities are preserved during rendering, and samples underwent manual inspection.

Questions for Authors

Q1. Evaluation on the 121 seed visualizations. We ran exploratory evaluations on the seed images and observed a similar overall trend. We did not report them formally because publication figures contain callouts, extraneous text, inconsistent backgrounds, cropping, and varying resolutions. Such artifacts may confound VLM evaluation even when humans understand the designs. Furthermore, seed annotations were intentionally non-exhaustive: uncertain elements were omitted because the goal was to extract design patterns, not create complete evaluation ground truth. Presenting these scores alongside standardized extension results would be methodologically questionable.

Q2. Human baseline on the extension set. A meaningful human baseline requires experts able to decompose diverse composite designs into mark groups, channels, repetition, and sharing relations. Recruiting experts with this breadth is difficult, and performance may vary with specialization and familiarity with particular designs. A baseline would depend heavily on annotator selection and protocol.

During construction, all three authors inspected potentially ambiguous cases and discussed them to reach consensus. We consider this a reasonable form of qualitative expert validation. It is not equivalent to an independent, statistically representative human baseline, but it provides human scrutiny of the structural annotations rather than relying solely on automatic generation.

Thank you for the authors' detailed responses. However, my primary concern is whether the annotated field-sharing relationships can be reliably inferred from the images alone. The authors clarify that the annotations are consistent with the underlying visualization specifications and have been manually inspected. However, this does not establish that these relationships are visually identifiable to independent observers. Without such validation, it remains unclear whether low field-sharing scores reflect limitations in structural understanding or ambiguities in the benchmark itself.

I also share mbYc's concerns regarding the omission of labels and the resulting visual ambiguities. In particular, the concrete examples identified in the subsequent discussion raise further questions about the prevalence of such cases and the reliability of the annotations as evaluation targets. Moreover, although the authors mention exploratory evaluations on real-world seed images, no quantitative results are provided to support the generalizability of the findings.

Since unresolved concerns about visual identifiability and evaluation validity directly affect the benchmark's central contribution and the reliability of its conclusions, I will maintain my original rating of 1.

AuthorsOpenReview ↗

Clarifying Visual Identifiability and Evaluation Validity

Clarifying Visual Identifiability and Evaluation Validity

We thank the reviewer for the follow-up. We agree that consistency with the underlying specifications and manual inspection do not constitute independent validation of visual identifiability. However, we would like to emphasize that many field-sharing relationships can indeed be inferred from observable visual structures, rather than relying on hidden specifications.

In composite visualizations, shared coordinates, spatial attachment, alignment, and repeated structures provide meaningful evidence of field correspondences. For example, in nested-01-013 and nested-03-016, boxplots and histograms are directly attached to parallel-coordinate axes, making their coordinate relationships visually apparent. In composite-01-013, corresponding facet identifiers and column-wise alignment support the interpretation of shared faceting. These relationships can be inferred from the visual composition itself, even without explicit field labels.

We acknowledge that visual evidence does not always uniquely determine the underlying field identity. However, the existence of alternative interpretations does not necessarily mean that the intended structural relationship cannot be reasonably inferred. Our goal is to evaluate such visually supported structural interpretations, rather than recover arbitrary hidden data provenance. We also recognize that some cases, such as composite-01-005, lack sufficient visual evidence and require correction. We will further clarify our annotation criteria and discuss the resulting limitations.

Regarding the real-world seed images, they were primarily used to derive structural patterns rather than for the main evaluation, partly because of backgrounds, callouts, and other visual artifacts. Our exploratory evaluations suggested similar trends, but we agree that without quantitative results, they cannot independently establish generalizability.

We acknowledge that independent human validation would provide stronger evidence of visual identifiability and will explicitly discuss this limitation. Nevertheless, we maintain that recovering field relationships from visual compositional cues is a meaningful and feasible diagnostic task, and that the identified ambiguities call for more careful annotation and interpretation rather than invalidate the overall evaluation objective.

Reviewer_mbYcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a benchmark - dataset and evaluation methodology - for evaluating VLMs in their understanding of linked multiple-view visualization designs. VLMs are evaluated through a standard decoding task, namely for each view in the visualization determine the type of graphical mark, data fields assigned to visual channels, and field-driven faceting (denoted as "repetition" in the paper). But the main emphasis of the paper is on evaluating VLM ability of identifying cross-view relationships, namely, data fields that appear in multiple views, potentially associated with different visual channels and mark types.

The main contribution of the paper is the construction of the dataset used for evaluation - starting from the VAID corpus, extracting a set of prototypical mark types, and then synthesizing a larger set of multiple-view designs that vary in structural complexity, e.g. some are single-view designs while in other designs individual fields are distributed amongst multiple views.

A secondary contribution is the evaluation of the latest frontier models, highlighting limitations in recognizing field correspondences for multiple-view designs.

Critical strengths

The main strength of the paper is the dataset, namely the steps involved in its construction. This consists of the proposed normalization scheme to ensure unambiguous descriptions of multiple-view visualizations, the treatment of discrete/continuous faceting, and ultimately the generation of visualization specifications, giving a fairly diverse set of multiple-view visualizations ideal for evaluating structural understanding.

Critical weaknesses

The central weakness of the paper is in the "presentation" of the visualizations. Specifically, no consideration is placed on how scales (field-to-channel mappings) are presented, e.g. for positional scales this would be axes, for color this would be color legends, etc.. Upon inspecting the generated visualization images, and given the details of dataset construction in the appendix, there are a number of issues:

  • Scales are presented without labels, namely, there is no display of field names.
  • Datasets are synthetically generated (to the best of my knowledge - in the appendix, there is reference to "synthetic rows").
  • No details are given on how axes are generated, namely, how the domain for the scale is chosen for continuous fields (e.g. min & max).

Combined, these issues make the problem of evaluating cross-view field correspondence ill-posed. Without scale labels, one must rely on the domain of the scales to determine whether two visual channels in different views encode the same data field. But just because two presented domains are the same does not imply a correspondence of fields - different fields can have the same domain. And the visualizations all tend to have very similar continuous domains ([0, 80]), likely a consequence of synthetic data generation.

A straightforward remedy is to modify the presentation - include scale labels in generated visualizations so that it is possible to unambiguously resolve fields of the dataset. Yet if this is done, I am concerned that the problem being studied now becomes one of recognizing and relating scale labels amongst multiple views - graphical marks become rather unimportant. I would anticipate the latest frontier models would perform well on this task, and so I am not sure how interesting this would be to study.

Questions for authors

Why is data aggregation excluded from the space of visualization designs in the benchmark? Aggregation is extremely common in multiple-view visualizations, in order to show data at different levels of detail. How would aggregation impact mark correspondence, and the resulting evaluation metrics?

6 comments
AuthorsOpenReview ↗

Clarifications on Field Identifiability, Rendering, and Aggregation

Critical Weaknesses

W1. Missing field and scale labels. CompVis-Bench does not ask models to recover original column names. Instead, it evaluates whether encodings in different mark groups refer to the same anonymous data field. Such correspondence can be supported by shared axes, spatial alignment, repetition, and other cross-view structures. Matching numerical scale domains is neither the only possible cue nor sufficient evidence of field sharing. In particular, two fields with identical domains are not automatically annotated as shared. We acknowledge that some configurations may be visually ambiguous, which is why we manually inspected potentially ambiguous field-sharing relationships, as explicitly described in the paper. However, the possibility of individual ambiguous cases does not establish that the benchmark is fundamentally ill-posed.

W2. Synthetically generated datasets. Synthetic data generation is a deliberate methodological choice, not an overlooked limitation. CompVis-Bench evaluates visualization structures and field correspondences, rather than the statistical properties or semantic meanings of underlying datasets. Synthetic data enable us to systematically generate diverse structural configurations while controlling unrelated variations in data content. The use of synthetic data does not inherently undermine the validity of structural understanding tasks. The reviewer's concern about similar numerical domains is a question of visual identifiability, rather than a fundamental problem with synthetic data generation itself.

W3. Scale-domain generation and axis sharing. Axis domains and other presentation properties are determined adaptively during rendering, rather than being explicitly specified as part of the structural annotations. Axis sharing, on the other hand, is generated by our randomized renderer as part of the visualization composition. These are distinct mechanisms: adaptive scale rendering determines how values are displayed, while axis sharing represents a structural relationship between visual components. Field-sharing ground truth is defined by the visualization specifications, not inferred from coincidentally matching numerical ranges. Therefore, similar scale domains do not imply shared fields, nor do they invalidate the structural annotations.

W4. Adding explicit field labels. We agree with the reviewer's own concern that displaying identical field names could reduce field correspondence identification to OCR and text matching. This would change the evaluation objective rather than simply remedy the problem. Our benchmark is designed to evaluate structural relationships without directly revealing field identities through text. We do not claim that all unlabeled configurations are unambiguous. However, the relevant question is whether field correspondences are visually identifiable, not whether explicit field names are provided. The absence of labels alone is insufficient to establish that the task is ill-posed.

Question for Authors: Aggregation

Q1. Why is aggregation excluded, and how would it affect field correspondence and evaluation metrics?

We would first like to clarify that aggregation is not excluded from our benchmark, nor is it treated as a special case in our evaluation.

Our benchmark evaluates whether visual components share the same underlying data field, regardless of whether their displayed values have undergone aggregation. For example, an aggregated bar chart and a discrete-distribution chart may share a positional axis. If this axis represents the same data field, the corresponding field-sharing relationship is annotated and evaluated, regardless of their different aggregation operations.

The important distinction is between observable field correspondence and hidden data-transformation provenance. When multiple original fields are combined through computation or data mining to produce a derived field, their computational dependencies generally cannot be inferred from the resulting visualization alone. We therefore do not require models to recover these hidden relationships.

Consequently, aggregation itself does not change the definition of our field-sharing metrics. Recovering computational dependencies between derived fields and their original inputs would require additional data-lineage annotations and transformation-aware metrics, which constitute a different evaluation objective.

The authors have not adequately addressed my concerns, and as such I will keep my score.

My main concern remains that the benchmark is poorly motivated. Simply put, there is no good reason to omit scale labels. For meaningful visual analysis, the end user (whether a person or VLM) needs to understand what aspects of the data that visual channels represent, and thus, scales need to be labeled. I fundamentally disagree that the visual identification of field correspondences is the "relevant question". I think what is most relevant is the comprehension of visualizations that adhere to basic design principles, namely, that the end user can unambiguously decode what (the data) is being visually encoded.

I also find it unusual to claim that "individual ambiguous cases does not establish that the benchmark is fundamentally ill-posed". This is especially problematic given that there seems to be some doubt about the number of cases that are actually ambiguous.

Even for visualizations that can be unambiguously resolved, the mechanism by which one would determine field correspondences can be extraordinarily tedious. For instance, for two bar charts that have distinct categorical fields, to determine whether they encode different quantitative fields, one must find a bar-to-bar correspondence between charts. This is the kind of task that one would never expect an end user to perform, and so again, I find the benchmark as a whole to be poorly motivated.

AuthorsOpenReview ↗

Clarification on Structural Understanding and Visual Design Relationships

We thank the reviewer for the further clarification. We believe our main disagreement concerns the level of visualization understanding that the benchmark aims to evaluate.

We agree that explicit field labels are important for communicating the semantic meaning of data to end users. However, our benchmark focuses on recovering higher-level visual design structures and the data relationships expressed through them, rather than identifying the semantic meaning of individual fields.

Importantly, many structural relationships can be inferred from visual design without knowing field names or exact scale values. For example, two charts sharing a common x-axis provide strong visual evidence of a shared positional field. Similarly, when pie charts are nested within a scatterplot, their placement at scatterplot marks suggests that the repeated pie charts are organized by the scatterplot's positional dimensions. These composition patterns are commonly used to express data relationships, and recognizing them does not necessarily require explicit field labels.

We also acknowledge that structural understanding is not always a process of recovering an absolutely unique interpretation from an image. For instance, does nesting a pie chart within a scatterplot necessarily mean that its position encodes data, or could it simply be a presentation choice? Likewise, does arranging multiple small multiples in a row necessarily indicate repetition over a data field, or could it be an arbitrary layout? Such uncertainties are inherent in interpreting visual designs, even when the underlying visualization specification is well-defined.

However, we do not believe that this uncertainty invalidates the meaningful structural relationships conveyed by well-designed visual compositions. Visualization design relies on established visual conventions through which spatial arrangements, shared coordinates, and repetition communicate relationships among data components. Our objective is to investigate whether VLMs can infer these plausible, design-grounded relationships from their visual manifestations.

We therefore do not claim that our benchmark provides an exhaustive or uniquely determined interpretation of every possible visualization. Instead, we construct annotations guided by visualization design principles and manually examine their visual recoverability, aiming to establish a benchmark that balances structural plausibility and perceptual identifiability.

We maintain that recovering design-implied structural relationships is a meaningful research objective in its own right. The absence of absolute certainty in every possible visual interpretation does not negate the value of evaluating whether models can recognize relationships conveyed through established composition patterns. Our benchmark is intended to measure this specific capability, rather than guarantee complete semantic understanding or eliminate all interpretive uncertainty inherent in visualization design.

Reviewer_Z2nqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether VLMs can understand the encoding structures of individual components in composite visualizations and the data-field sharing relationships between components. The authors introduce CompVis-Bench, which uses a normalized structural representation and a structural reconstruction task to evaluate component identification, local encoding recovery, cross-component field correspondence recovery, and complete reconstruction at different levels. The dataset includes 121 real-world visualization seed examples selected from VAID and reannotated, along with 1,600 synthetic extension examples constructed through structural extension, randomized rendering, and manual verification. The authors evaluate 12 large vision-language models on the extension set and find that models are relatively good at recovering local components and encodings, but still have substantial difficulty recovering globally consistent cross-component field relationships.

Critical strengths
  1. It focuses on structural understanding of composite visualizations, using a normalized representation to explicitly describe local encodings and cross-component field relationships, and evaluates these through metrics at multiple levels. This design helps identify gaps in model capabilities.

  2. The experiments cover multiple model families and reveal a clear gap between local encoding recovery and cross-component field correspondence recovery. They also show that models can recover local visual encodings but still have limitations in understanding global relationships across components.

Critical weaknesses
  1. The evaluation is based on synthetic extension examples whose structural annotations and mock-data scripts were generated with assistance from GPT-6-Astra, which is also one of the evaluated models. Furthermore, manual verification alone is insufficient to establish that the synthetic examples are representative of the distribution of real-world composite visualizations.
Questions for authors

As noted in the weaknesses, my question is whether these factors could affect model scores, relative rankings, and the applicability of the main conclusions to real-world settings.

2 comments
AuthorsOpenReview ↗

Clarifications on Dataset Construction and Generalizability

We thank the reviewer for the positive assessment and for raising these important concerns regarding potential model bias and real-world generalizability.

Critical Weaknesses

W1. Model-assisted generation and dataset representativeness. We would like to clarify the actual construction process, which involves several stages rather than directly using GPT-6-Astra to generate the final annotations and images.

First, GPT-6-Astra produces an initial textual specification through rule-guided structural extension, based on a selected seed annotation and a corresponding prompt. This specification is then processed by our randomized renderer, which independently generates mock data and renders the visualization image. Finally, the authors inspect the rendered visualization and revise annotations that contain information not reliably recoverable from the image.

Thus, the final annotations and images are not direct outputs of GPT-6-Astra. The initial model-generated specifications can undergo substantial changes through rendering and manual correction. Importantly, mock data are generated by the randomized renderer, not by GPT-6-Astra. We apologize for the inaccurate description of mock-data generation in the paper and will correct it.

We acknowledge that using a model outside the evaluated set for structural extension would provide stronger independence. Nevertheless, the multi-stage construction process, particularly randomized rendering and manual correction, reduces direct dependence on the initial model outputs.

Regarding representativeness, our extension set is grounded in structural patterns extracted from 121 real-world VAID examples. Its purpose is not to reproduce the exact distribution of real-world composite visualizations, but to systematically extend their structural configurations for controlled evaluation.

Questions for Authors

Q1. Potential effects on model scores, rankings, and real-world applicability. We agree that the distribution of generated structures may affect absolute scores and relative model rankings. Although our multi-stage construction process reduces dependence on GPT-6-Astra, we do not claim that it completely eliminates potential generation bias.

However, all 12 evaluated models are tested against the same structural ground truth, which is reviewed and corrected by the authors rather than directly adopted from the initial model-generated specifications. This provides a consistent basis for comparing structural reconstruction capabilities under controlled conditions.

We also conducted exploratory evaluations on the real-world seed images and observed a similar overall performance trend. We chose not to report these results formally because the source images contain uncontrolled artifacts, such as callouts, cropping, and inconsistent visual quality, while their annotations were designed for extracting structural patterns rather than providing exhaustive evaluation ground truth.

We agree that further evaluation on standardized real-world composite visualizations would strengthen the generalizability of our findings. Our current conclusions primarily concern structural understanding under controlled evaluation conditions, rather than claiming that the observed model rankings necessarily hold across all real-world settings.

Thank you for the clarification. It largely addresses my original concerns, and I will keep my original rating.

I have one follow-up question. Would it be feasible to run a small blinded assessment in which independent readers judge field sharing from the images alone, and report how often they agree and how often they find the relationship indeterminate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.