acceptodds
Under review as a conference paper at ICLR 2027

ChartGalaxy++: A Richly Annotated Dataset for Chart Understanding and Generation

26 views

Abstract

Infographic charts communicate data through cohesive compositions of visual elements such as charts, text, and images. Existing infographic chart datasets primarily annotate individual elements with bounding boxes and category labels. Such element-level annotations do not capture hierarchical and spatial relationships, limiting their use for understanding how elements are organized and related within an infographic and for generating well-organized designs. To address this limitation, we introduce ChartGalaxy++, a richly annotated dataset of 217,195 infographic charts. At the core of this dataset is a scene graph representation, in which nodes represent visual elements and semantic groups, and edges encode their hierarchical and spatial relationships. For real infographics, we construct scene graphs by detecting and progressively grouping visual elements; for synthetic ones, we derive them directly from the generation process. We demonstrate the utility of ChartGalaxy++ through three applications: 1) image-to-scene-graph prediction, 2) scene-graph-augmented infographic question answering, and 3) evaluation of scene graph preservation in image generation. Together, these applications highlight the value of ChartGalaxy++ for advancing chart understanding and generation.

Primary area:
datasets and benchmarks
Keywords:
Infographic Chart, Chart Understanding, Chart Generation, Dataset, Scene Graph

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UvgdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ChartGalaxy++, a dataset of 217,195 infographic charts built on ChartGalaxy, each annotated with a scene graph that groups visual elements into semantic groups encoding their hierarchical and spatial relationships. Real charts are annotated by an OCR/detection pipeline combined with Gemini 3.8 Flash for attributes and grouping, while synthetic ones are parsed from their source SVGs, and the annotations of a 1,000-image test set are manually verified. The authors validate the dataset through three applications: (1) a fine-tuned Qwen3.5-4B image-to-scene-graph model that outperforms ten LVLMs; (2) a QA benchmark where adding annotated scene graphs to the prompt improves accuracy by 8.9 points on average; and (3) an evaluation of scene graph preservation in 11 image-generation models.

Critical strengths

1. Large-scale semantic hierarchy annotations for infographic charts. ChartGalaxy++ annotates 217K infographic charts with semantic groups and their hierarchy, not just individual elements. Earlier structural datasets either cover only plain charts (VisText, VisAnatomy) or group elements by layout without data semantics (Shi et al., 2023). This is the main reason I lean toward acceptance, since the dataset is directly useful for training and evaluating structure-aware chart models.

2. Broad experimental coverage. The paper evaluates the dataset in three settings and compares a wide range of models: 10 LVLMs for scene graph prediction and QA, and 11 image-generation models for scene graph preservation. This gives a useful picture of where current models stand on infographic structure, for example that even GPT-6 Astra reaches only 50.3 hierarchical F1.

Critical weaknesses

1. The experiments do not show that the dataset improves models beyond its own distribution. The image-to-scene-graph comparison in Sec. 5.1 pits a model fine-tuned on ChartGalaxy++ against zero-shot LVLMs on an in-distribution test set annotated under the same conventions. Outperforming them is expected and does not by itself show the value of the dataset. The QA results in Sec. 5.2 rely on scene graphs produced by the annotation pipeline rather than predicted by the fine-tuned model, and are reported only on the authors' own 1,266-question benchmark. Table 3 further suggests that most of the QA gain comes from node text with digits rather than relationships, and the concise node labels may already state which entity a value belongs to, which would turn within-group questions into a lookup. This matters because these two applications are the main evidence that the dataset helps chart understanding. The authors should evaluate the fine-tuned model on out-of-distribution infographics, report QA results using its predicted scene graphs on existing benchmarks such as InfoChartQA, InfographicVQA, or ChartQAPro, and include a flat OCR-plus-box baseline and models trained on alternative data (e.g., InfoDet or only the real or synthetic subset).

2. The annotations come from a single model, and their reliability is not well established. For real charts, element types, descriptions, and the entire hierarchy are produced and verified by Gemini 3.8 Flash, and the training set receives no human verification. The audit covers only a curated test set and corrects pipeline outputs rather than annotating independently, so the reported error rates likely overestimate overall quality. The grouping also follows detailed conventions (Prompt B.6), and it is unclear whether the baselines in Sec. 5.1 are given them. If not, they are scored against conventions they never saw, and since hierarchical F1 only credits exact direct parents, a single extra or missing group level makes all edges below it count as errors. This matters because every result in the paper rests on these annotations. The authors should audit a random training sample, report the pipeline's own score on the test set before correction, and evaluate baselines with the grouping rules provided.

Questions for authors

Clarifications

  1. Are the baselines in Sec. 5.1 given the grouping rules in Prompt B.6, or only the scene graph schema? If only the schema, how much of the hierarchical F1 gap remains when the rules are provided?
  2. Do the concise QA labels (Appendix E.3) name the entity a value belongs to? If so, the association is available from the label text alone, and the QA gain cannot be attributed to the hierarchy. Please provide a complete scene graph input for one QA question.
  3. In Table 3, does "relationships" include parent references or only spatial relations? If it includes parents, "Best − relationships" (85.6) is effectively a flat baseline.
  4. Of the 1,000 charts in the QA benchmark, how many belong to the human-verified test set, and how many are real versus synthetic?
  5. How are SVG <g> groups in synthetic charts mapped to the seven semantic group types, and are the resulting hierarchies consistent with the conventions in Prompt B.6?

Additional results that could change my assessment

  1. QA accuracy with scene graphs predicted by the fine-tuned model, ideally also on an existing benchmark such as InfoChartQA, InfographicVQA, or ChartQAPro.
  2. QA accuracy with neutral node labels (e.g., text_12) and, if not covered by Table 3, with a flat OCR-plus-box input.
  3. The pre-correction pipeline output on the test set scored with the Table 1 metrics, and error rates from an audit of a random training sample.
  4. Table 1 results split by real and synthetic test charts.

Suggestions for future work (not affecting my score)

  1. Training an image-generation model conditioned on scene graphs, rather than only evaluating existing models.
  2. Replacing Gemini with an open model in the annotation pipeline, to assess how much the dataset depends on a single proprietary model.
7 comments
AuthorsOpenReview ↗

Response to Reviewer Uvgd (1/5)

We sincerely thank you for your detailed review and thoughtful suggestions. We have carefully addressed the concerns regarding the generalization of ChartGalaxy++ beyond its own distribution, the reliability of the annotations, the settings of the baselines and the QA ablation, the construction of scene graphs for synthetic charts, the training of image-generation models, and the use of an open model in our annotation pipeline.

Q1: The experiments do not show that the dataset improves models beyond its own distribution.

A1: Per your suggestions, we have added evaluations of scene graphs predicted by our fine-tuned model on out-of-distribution infographics and assessed their utility for QA on four existing benchmarks through comparisons with additional baselines (Appendix E.7). We have also added an ablation that separates the contributions of semantic labels and explicit relationships (Appendix E.4).

  1. Quality of predicted scene graphs on out-of-distribution (OOD) infographics.

We randomly sample 300 charts from three existing benchmarks, 100 each from InfoChartQA, InfographicVQA, and ChartQAPro, and manually verify the scene graphs predicted by our fine-tuned model following the assessment protocol in Appendix B.3, covering 33,271 non-root nodes.

MetricPredicted, OOD chartsPipeline, ChartGalaxy++ test set (Appendix B.3)
Element precision93.98%99.81%
Element recall91.05%99.78%
Mean box IoU0.92640.9976
Text transcription errors1.53%0.39%
Numeric transcription errors1.48%0.12%
Description / role errors5.31%3.08%

The predicted scene graphs recover most elements on OOD charts, with element precision and recall above 91% and a mean box IoU of 0.9264. Text and numeric transcription errors remain below 2% (1.53% and 1.48%), and the description or role error rate is 5.31%. These results provide evidence that the model can extract chart elements beyond ChartGalaxy++.

  1. QA performance of predicted scene graphs on existing benchmarks.

We evaluate Qwen3.8-27B's QA performance on four existing benchmarks, InfoChartQA, InfographicVQA, ChartQAPro, and CharXiv, as well as on our benchmark (Appendix E.7). We compare four settings: (1) direct prompting, (2) flat OCR and bounding boxes, (3) elements predicted by the detector released with InfoDet, and (4) scene graphs predicted by our fine-tuned image-to-scene-graph model. All settings use Qwen3.8-27B as the QA model. The scene graphs in this experiment are model predictions, rather than outputs of the annotation pipeline.

We report ANLS on InfographicVQA and accuracy on the other benchmarks.

InputOursInfoChartQAInfographicVQAChartQAProCharXiv
Direct prompting75.182.592.667.692.0
+ Flat OCR + boxes81.882.092.367.991.9
+ Elements from InfoDet-trained detector77.282.592.567.891.8
+ Scene graphs predicted by our model83.883.193.268.192.8

Among the evaluated settings, predicted scene graphs achieve the highest QA accuracy on all five benchmarks. On our benchmark, they improve accuracy by 8.7 percentage points over direct prompting and by 2.0 points over flat OCR + boxes. On the four external benchmarks, they improve accuracy over direct prompting by 0.6, 0.6, 0.5, and 0.8 points, respectively. In comparison, the flat OCR + boxes input and the InfoDet-trained detector yield gains of at most +0.3+0.3 and +0.2+0.2 points. These external gains are modest, but they provide evidence that the utility of predicted scene graphs extends beyond our own benchmark. Their smaller magnitude may reflect the broader mix of questions in these benchmarks, whereas ours focuses on questions requiring element associations. We also observe larger gains over direct prompting on question types that depend on such associations, such as ranking in InfoChartQA (+1.7+1.7) and counting in InfographicVQA (+1.5+1.5) (Appendix E.7). We are also training the image-to-scene-graph model on the real and synthetic subsets separately. We will report the results during the discussion period if they become available.

AuthorsOpenReview ↗

Response to Reviewer Uvgd (2/5)

Q1 (continued)

A1 (continued): 3. Contributions of semantic labels and explicit relationships to QA performance.

We agree that concise node labels can convey entity associations and make some within-group questions easier to answer through lookup. For example, the shared entity name in value_label_land_bank_of_the_philippines and data_item_land_bank_of_the_philippines suggests their association. However, the scene graph structure makes this association explicit through its relationships. To separate these contributions, we have added an ablation that replaces semantic labels with neutral labels (node_i), both with and without relationships (Appendix E.4).

SettingLabelsRelationshipsAcc. (%)vs. Full scene graph
Full scene graphSemanticwith86.9—
Full scene graph − relationshipsSemanticwithout85.6−1.3
Full scene graph, neutral labelsNeutralwith86.5−0.4
Full scene graph − relationships, neutral labelsNeutralwithout83.4−3.5

With semantic labels, adding relationships improves accuracy by 1.3 percentage points (85.6% to 86.9%), since the labels still convey part of the hierarchy. With neutral labels that carry no such information, the gain increases to 3.1 percentage points (83.4% to 86.5%). Semantic labels identify the entities to which elements belong and can therefore convey grouping information even when explicit relationships are removed. These results suggest that relationships benefit QA, and semantic labels partly compensate for their absence.

Q2: The annotations come from a single model, and their reliability is not well established.

A2: Per your suggestion, we have evaluated the pre-correction pipeline outputs on the test set with the Table 1 metrics and manually audited 500 randomly sampled training charts. We also clarify that the baselines in Sec. 5.1 are already provided with the grouping rules.

  1. Pre-correction pipeline outputs on the test set.

Our annotation-quality assessment in Appendix B.3 records the errors in the pipeline outputs before correction. We further score these pre-correction outputs against the human-corrected test set annotations with the Table 1 metrics. All scores below are on a 0–100 scale.

Test subsetSizeOverallTypeBoxTextColorDescriptionHierarchicalSpatial
All1,00099.1699.9599.7599.7099.8199.5799.3098.75
Real50098.7999.9199.5299.6099.8199.5599.2498.15
Synthetic50099.5299.9899.9199.7999.8199.5899.4599.35

As shown in the table above, the pre-correction pipeline outputs achieve an overall node-level F1 of 99.16 and a hierarchical F1 of 99.30, with similar hierarchical scores for real and synthetic charts (99.24 and 99.45).

  1. Audit of randomly sampled training charts.

We randomly sample 500 charts from the 216,195 training charts and manually assess their annotations following the protocol in Appendix B.3.

MetricExisting test-set assessment (1,000 charts)Random training-sample audit (500 charts)
Element precision99.81%99.80%
Element recall99.78%97.75%
Mean box IoU0.99760.9929
Text transcription error0.39% (134/34,242)1.29% (171/13,256)
Numeric transcription error0.12% (27/21,856)0.12% (10/8,485)
Description / role error3.08% (3,053/99,010)5.39% (2,312/42,896)
Incorrect parent assignment0.010% (10/99,010)0.040% (17/42,896)

The training sample retains high element precision and box IoU, with the same numeric transcription error rate as the test set (0.12%). Incorrect parent assignments remain rare (0.040%). However, element recall is slightly lower (97.75% vs. 99.78%). Text transcription errors increase from 0.39% to 1.29%, and description/role errors from 3.08% to 5.39%.

  1. Grouping rules provided to the baselines.

We would like to clarify that the baselines in Sec. 5.1 are provided with both the scene graph schema and the grouping rules in Prompt B.6. We have revised Sec. 5.1 to state this explicitly, and added the baseline prompt to Appendix D.2.

Q3: Are the baselines in Sec. 5.1 given the grouping rules in Prompt B.6, or only the scene graph schema? If only the schema, how much of the hierarchical F1 gap remains when the rules are provided?

A3: We would like to clarify that the baselines are provided with both the scene graph schema and the grouping rules in Prompt B.6. We have revised Sec. 5.1 to state this explicitly, and added the baseline prompt to Appendix D.2.

AuthorsOpenReview ↗

Response to Reviewer Uvgd (3/5)

Q4: Do the concise QA labels (Appendix E.3) name the entity a value belongs to? If so, the association is available from the label text alone, and the QA gain cannot be attributed to the hierarchy. Please provide a complete scene graph input for one QA question.

A4: Yes, as described in Appendix E.3, concise node labels summarize node descriptions and grouping context, and can therefore identify the entity to which a value belongs. We agree that the QA gain cannot be attributed solely to the hierarchy. To separate the contributions of semantic labels and explicit relationships, we added a neutral-label ablation (Appendix E.4; see Q1). With neutral labels, accuracy remains 86.5%, compared with 86.9% with semantic labels. Removing relationships under the same neutral-label setting reduces accuracy to 83.4%. This 3.1-point difference supports a contribution from both hierarchical and spatial relationships.

Due to the response length limit, we present selected parts of the scene graph JSON inputs for the four examples in Appendix E.1. The complete scene graph inputs for all QA questions are available in our GitHub repository (https://github.com/ChartGalaxyPP/ChartGalaxyPlusPlus/releases/download/v1.0/qa.tar.gz).

Q5: In Table 3, does "relationships" include parent references or only spatial relations? If it includes parents, "Best − relationships" (85.6) is effectively a flat baseline.

A5: In Table 3, "relationships" includes both hierarchical parent references and spatial relations. Although "Best − relationships" (85.6%) is structurally flat, it is not equivalent to a flat baseline with neutral labels, because its semantic labels still implicitly encode part of the hierarchy. To separate these sources of information, we added a flat baseline with neutral labels (node_i), retaining node types, bounding boxes, and text (Appendix E.4; see also our response to Q1). This baseline achieves 83.4% accuracy. Adding relationships while keeping neutral labels raises accuracy to 86.5% (+3.1 points); restoring both relationships and semantic labels yields 86.9% (+3.5 points).

Q6: Of the 1,000 charts in the QA benchmark, how many belong to the human-verified test set, and how many are real versus synthetic?

A6: The QA benchmark contains 1,000 charts: 500 real and 500 synthetic. Of these, 145 (14.5%) also belong to the human-verified test set, including 66 real and 79 synthetic charts.

Q7: How are SVG <g> groups in synthetic charts mapped to the seven semantic group types, and are the resulting hierarchies consistent with the conventions in Prompt B.6?

A7: We modify the chart-generation code to explicitly label each SVG <g> group with one of the seven semantic group types. The renderer records these labels and the group nesting in the SVG, from which we construct the scene graph. The resulting hierarchies follow the same annotation conventions as those used for real charts in Prompt B.6.

Q8: QA accuracy with scene graphs predicted by the fine-tuned model, ideally also on an existing benchmark such as InfoChartQA, InfographicVQA, or ChartQAPro.

A8: We evaluated QA using scene graphs predicted by our fine-tuned model on our benchmark and four existing benchmarks: InfoChartQA, InfographicVQA, ChartQAPro, and CharXiv. These predicted scene graphs yield the highest QA scores among the evaluated settings on all five benchmarks. Please refer to our response to Q1 for the detailed results and baseline comparisons.

Reviewer_eE8oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a dataset with scene-graph annotation of infographic charts, capturing the hierarchical and spatial relation among elements. The authors then apply this dataset to three use cases: training an infographic-to-scene-graph model, scene-graph-augmented infographic question answering, and evaluating scene graph preservation of image generation models, contributing to infographic chart understanding and generation.

Critical strengths
  1. The authors built an automated annotation pipeline with iterative refinement and complementary validation methods, scaling the annotation to >200k images with manageable cost, while showing high annotation quality in human audit. This is the most interesting part to me, and can inspire benchmark curation in other domains, so I lean toward acceptance.
  2. The motivation of representing infographic charts as scence graphs is clear; the three use cases of the dataset are generally convincing.
Critical weaknesses
  1. The ablation results in Table 3 seem to indicate that most of the improvement is from the text with digits, while the relationship only contributes around 1 percent. This somewhat undermines the claim that scene graph annotation advances previous work by capturing the hierarchical and spatial relations among elements.
  2. For infographic QA, while Section 5.2 shows the benefit of scene graph representation, its relative effectiveness over alternative structured representation, like SVG, is not clear. It would help to add a baseline of generating the SVG from the real infographic chart and provide it to the QA model.
Questions for authors
  1. Clarification: Can you provide more details of the ablation experiment settings of Table 3, and add significance tests?
  2. Clarification: How does scene graph representation compare to SVG for improving QA on real infographic charts?
5 comments
AuthorsOpenReview ↗

Response to Reviewer eE8o (1/2)

We appreciate your valuable feedback and suggestions. Per your suggestions, we have carefully addressed the concerns regarding the contribution of relationships in the Table 3 ablation, the comparison between scene graphs and SVGs as structured inputs for QA, and the detailed settings and significance tests of the ablation.

Q1: The ablation results in Table 3 seem to indicate that most of the improvement is from the text with digits, while the relationship only contributes around 1 percent. This somewhat undermines the claim that scene graph annotation advances previous work by capturing the hierarchical and spatial relations among elements.

A1: We agree that text with numerical values brings a large gain in Table 3, while relationships contribute about 1 point. To further analyze the reason, we have added an ablation study in Appendix E.4, using the same 1,266 QA questions and metric. The results show that the semantic labels, which name the entity each element belongs to, partly compensate for the removed relationships.

SettingLabelsRelationshipsAcc. (%)
Full scene graphSemanticwith86.9
Full scene graph − relationshipsSemanticwithout85.6
Full scene graph, neutral labelsNeutralwith86.5
Full scene graph − relationships, neutral labelsNeutralwithout83.4
Flat OCR + boxesNonewithout81.8

As shown in the table above, with semantic labels, removing relationships lowers accuracy by only 1.3 points (86.9% → 85.6%), since the labels still convey part of the hierarchy. With neutral labels (node_i) that carry no such information, the same removal lowers accuracy by 3.1 points (86.5% → 83.4%). These results suggest that relationships benefit QA, and semantic labels partly compensate for their absence.

Moreover, we further exploit the scene graph to improve QA through structure-aware subgraph retrieval (Sec. 5.2 and Appendix E.6). Providing the full scene graph adds about 8,000 tokens per question on average, accounting for 76.9% of the input tokens, while the model cites only 5.7% of the scene graph nodes as evidence. We therefore use the hierarchy to provide only the part of the scene graph that a question needs (Sec. 5.2 and Appendix E.6). Qwen3.8-27B first locates the nodes relevant to the question, and we then complete them into a subgraph by following the hierarchy. Answering with this subgraph instead of the full scene graph raises accuracy from 86.9% to 90.1%.

Q2: For infographic QA, while Section 5.2 shows the benefit of scene graph representation, its relative effectiveness over alternative structured representation, like SVG, is not clear. It would help to add a baseline of generating the SVG from the real infographic chart and provide it to the QA model.

A2: Per your suggestion, we have added an SVG-augmented baseline in Appendix E.5. We evaluate on the 500 real infographic charts and their 728 questions in the QA benchmark. Our fine-tuned model (Sec. 5.1) predicts scene graphs directly from the chart images, while GPT-6.1 Sol [1] reconstructs SVGs from the same images. Qwen3.8-27B performs QA using either the predicted scene graph or the reconstructed SVG as structured input, following the setting in Sec. 5.2.

Structured inputAcc. (%)
Predicted scene graph87.23
Reconstructed SVG82.28

As shown in the table above, predicted scene graphs outperform reconstructed SVGs by 4.95 percentage points. Manual inspection finds errors, including incorrect text and element associations, in 78.6% of the reconstructed SVGs. On questions associated with these charts, the accuracy gap widens to 6.26 percentage points (86.00% with scene graphs versus 79.74% with reconstructed SVGs). This pattern suggests that reconstruction errors can limit the usefulness of SVG inputs for QA. These findings highlight the difficulty of faithfully reconstructing SVGs from real infographic charts and suggest that reconstruction errors can reduce their usefulness for downstream QA.

Accordingly, these results demonstrate the effectiveness of our predicted scene graphs relative to the tested SVG reconstruction baseline for QA on real infographic charts.

References

[1] OpenAI, "GPT-6.1 Sol," Model documentation, 2026.

AuthorsOpenReview ↗

Response to Reviewer eE8o (2/2)

Q3: Clarification: Can you provide more details of the ablation experiment settings of Table 3, and add significance tests?

A3: Thank you for this suggestion. We have added the detailed ablation settings and significance tests to Appendix E.4.

  1. Ablation settings.

All settings in Table 3 use Qwen3.8-27B (checkpoint 1d4bf0f) served with vLLM 0.28.0, with thinking enabled at low reasoning effort, greedy decoding (temperature 0), and a maximum of 16,384 output tokens. They use the same 1,266 questions, chart images, answer-format instructions, guided system instruction (Appendix E.3), and scoring rules (Appendix E.2). Only the scene graph input differs.

The rows add components cumulatively, from structure to content:

  • + node types & bounding boxes: provides each node's type and bounding box, using schema labels such as <text 4>, without relationships or text content.
  • + relationships: adds parent references and spatial relationships using the same schema labels.
  • + labels: replaces the schema labels with concise semantic labels.
  • + text without digits: adds node text while excluding numerical values from value-bearing nodes, such as value labels and axis tick labels. These values are removed together with their units and currency symbols. Years, category names, ranks, and ordinal labels (e.g., "No. 3") are retained.
  • + text with digits (full scene graph): restores the excluded quantitative values, providing the complete node text.

For the additional ablations below, removing relationships leaves the node information unchanged, while the neutral-label variants replace semantic labels with identifiers such as node_i. The flat OCR + boxes baseline retains all recognized text, including numerical values, and its bounding boxes, but excludes non-text nodes and graph relationships.

  1. Significance tests.

Each setting is run three times, and Table 3 reports the mean accuracy. For each setting, we first average correctness for each question across the three runs. We then compute nominal pointwise 95% confidence intervals and two-sided p-values for the mean accuracy differences using t-tests with finite-sample-corrected, chart-clustered robust standard errors. This accounts for the 1,266 questions grouped within 1,000 charts. Confidence intervals are unadjusted, whereas p-values are Holm-adjusted.

ComparisonDiff. (pts)95% CIHolm pp
Full scene graph vs. no scene graph+10.02[8.10, 11.94]3.17e-15
Full scene graph vs. flat OCR + boxes+5.06[3.31, 6.80]8.57e-8
Full scene graph vs. Full scene graph − relationships+1.26[0.10, 2.43]0.0667
Full scene graph vs. Full scene graph, neutral labels+0.39[−1.08, 1.87]0.5983
Full scene graph, neutral labels vs. Full scene graph − relationships, neutral labels+3.13[1.85, 4.41]7.16e-6
Full scene graph vs. Full scene graph − relationships, neutral labels+3.53[2.09, 4.97]7.16e-6

More details of settings and significance tests are provided in Appendix E.4.

Q4: Clarification: How does scene graph representation compare to SVG for improving QA on real infographic charts?

A4: Model-predicted scene graphs outperform reconstructed SVGs by 4.95 percentage points in QA accuracy on real infographic charts. Our error analysis suggests that SVG reconstruction errors contribute to this gap. Please refer to our response to Q2 for details.

AuthorsOpenReview ↗

Looking Forward to Further Feedback

Thank you again for your thoughtful reviews and constructive feedback. We sincerely appreciate the time and effort you have devoted to evaluating our work.

We have carefully addressed all of the concerns raised in the reviews through additional ablations on the contribution of relationships, additional experiments comparing scene graphs with SVGs for QA, and detailed ablation settings with significance tests in our rebuttal. We would greatly appreciate it if you could take a moment to review our response and let us know whether they have adequately addressed your concerns.

If there are still any remaining questions or aspects that you feel require further evidence or clarification, we would be more than happy to provide additional experiments or explanations during the discussion period. Your feedback is extremely valuable in helping us improve the paper.

Thank you again for your time and consideration.

Reviewer_CdVWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ChartGalaxy++, which adds scene graph annotations to 217,195 infographic charts drawn from ChartGalaxy. In each graph, leaf nodes are individual text, image or shape elements with their attributes, and internal nodes group these elements into semantic types. Edges capture both the parent–child hierarchy and spatial relations derived from bounding boxes. Annotations for real charts are produced by a VLM pipeline with iterative verification. Annotations for synthetic charts are extracted directly from the SVGs that generated them. The dataset is used for scene graph prediction, scene-graph-augmented infographic question answering, and serves as a benchmark for image generators, scored on how well their outputs preserve an input scene graph.

Critical strengths

The dataset is likely to be a useful resource for the community. Existing infographic datasets annotate elements in isolation, while ChartGalaxy++ explicitly records how elements group into data items, series, legends and axes, at a scale beyond prior structural chart datasets.

Critical weaknesses
  1. Limited methodological novelty. The contribution is mainly data engineering rather than new methodology. The images come from ChartGalaxy, and the annotation pipeline chains existing tools with standard programmatic checks and VLM self-verification. Representing visual layouts as hierarchical graphs is also well established, as the related work section acknowledges.

  2. The value of the data for training is not tested. No experiment shows that training on ChartGalaxy++ improves chart understanding. For example, the paper does not test whether multi-task training improves results on standard chart understanding datasets (like CharXiv or InfographicsVQA/ChartQA), or whether graphs predicted by the fine-tuned model improve downstream QA. Training generators on the data is left to future work. As a result, the dataset’s value as training data is argued but not demonstrated.

  3. The QA results do not show that structure matters. In the Table 3 ablation, the largest gain comes from adding text with digits, which points to OCR content rather than structure. Adding relationships gives very minor gains. A flat OCR plus boxes baseline without hierarchy is needed to isolate the contribution of the graph structure.

  4. Venue fit. This is primarily a dataset and benchmark paper, and its contribution to representation learning is limited. Dataset papers can belong at ICLR, but in its current form this one would fit better at a venue such as the NeurIPS Datasets and Benchmarks track.

Questions for authors

Please address the weaknesses items above

4 comments
AuthorsOpenReview ↗

Response to Reviewer CdVW (1/3)

We truly appreciate your thoughtful feedback and constructive suggestions. In our responses below, we carefully address the concerns regarding the methodological novelty of our work, the value of ChartGalaxy++ as training data, the contribution of the graph structure to QA, and the fit of our work to ICLR.

Q1: Limited methodological novelty. The contribution is mainly data engineering rather than new methodology. The images come from ChartGalaxy, and the annotation pipeline chains existing tools with standard programmatic checks and VLM self-verification. Representing visual layouts as hierarchical graphs is also well established, as the related work section acknowledges.

A1: Thanks a lot for raising this concern. We agree that our annotation pipeline builds on existing tools and that hierarchical representations are well established. However, we would like to clarify the distinction between prior scene-tree-based annotations and our scene graph annotations. Our primary contribution is ChartGalaxy++, which adds large-scale, design-informed structural annotations to ChartGalaxy images for infographic understanding and generation.

VisText [1] and VisAnatomy [2] provide scene-tree-based annotations for 12,441 Vega-Lite-generated charts and 942 SVG-based plain charts, respectively. These trees encode hierarchical organization through parent-child relationships. ChartGalaxy++ extends this representation to scene graphs by adding explicit spatial relationships among visual elements. Their extraction is guided by the CRAP design principles [3] (Contrast, Repetition, Alignment, and Proximity), so that the annotations capture layout relationships beyond hierarchical membership. The contribution is therefore not simply a larger collection of hierarchical annotations, but richer structural supervision that represents both element organization and design-informed spatial relationships.

ChartGalaxy++ provides scene graph annotations for 217,195 infographic charts, including 56,744 real infographic charts whose structures are recovered directly from raster images. Although the images come from ChartGalaxy, the added scene graph annotations provide explicit supervision for learning hierarchical and spatial relationships among visual elements. They make both element grouping and design-informed layout relationships available as learning targets, supporting structure-aware representations that capture how visual elements are organized and related. ChartGalaxy++ therefore contributes to representation learning by providing structural supervision at scale for infographic understanding and generation.

The annotation pipeline enables this data contribution by separating element annotation, hierarchy construction, and CRAP-guided spatial relationship extraction, allowing intermediate outputs to be checked and refined. On the 500 real test charts, it achieves a hierarchical F1 of 99.24 before manual correction, compared with 57.68 when the same LVLM generates scene graphs directly. This 41.56-point improvement demonstrates the effectiveness of the staged pipeline for hierarchy recovery, rather than the novelty of its individual tools.

To further demonstrate the downstream utility of ChartGalaxy++'s structural annotations, during the rebuttal we also developed and evaluated a structure-aware subgraph retrieval method for chart QA. The method identifies question-relevant nodes and expands them along the hierarchy to preserve their structural context. Using Qwen3.8-27B as the QA model in both settings, we obtain 90.1% accuracy with the retrieved subgraphs, 3.2 percentage points above the 86.9% obtained with the full scene graph. This result illustrates how the dataset's explicit structural annotations can support retrieval and reasoning for chart QA.

We have revised the Introduction, Related Work, Sec. 4, and Sec. 5 to clarify these contributions: extending scene-tree-based annotations with CRAP-guided spatial relationships, providing the resulting scene graphs at scale for infographic charts, and enabling their construction directly from raster images.

References

[1] B. Tang, et al., "VisText: A Benchmark for Semantically Rich Chart Captioning," ACL, 2023.

[2] C. Chen, et al., "VisAnatomy: An SVG Chart Corpus with Fine-Grained Semantic Labels," IEEE TVCG, 2026.

[3] R. Williams, "The Non-Designer's Design Book," 4th ed., Peachpit Press, 2015.

AuthorsOpenReview ↗

Response to Reviewer CdVW (2/3)

Q2: The value of the data for training is not tested. No experiment shows that training on ChartGalaxy++ improves chart understanding. For example, the paper does not test whether multi-task training improves results on standard chart understanding datasets (like CharXiv or InfographicsVQA/ChartQA), or whether graphs predicted by the fine-tuned model improve downstream QA. Training generators on the data is left to future work. As a result, the dataset's value as training data is argued but not demonstrated.

A2: Thanks a lot for this suggestion. We have added an experiment to assess whether scene graphs predicted by our fine-tuned model (Sec. 5.1) improve QA performance beyond ChartGalaxy++ (Appendix E.7).

We use the predicted scene graphs as structured input for QA and evaluate QA performance on our benchmark and four existing benchmarks: InfoChartQA, InfographicVQA, ChartQAPro, and CharXiv. For comparison, we use element-level predictions from a detector trained on InfoDet [1], an infographic dataset with element-level annotations. All settings use Qwen3.8-27B as the QA model.

InputOursInfoChartQAInfographicVQAChartQAProCharXiv
Direct prompting75.182.592.667.692.0
+ Elements from the InfoDet-trained detector77.282.592.567.891.8
+ Scene graphs predicted by our model83.883.193.268.192.8

Using our predicted scene graphs achieves the highest QA accuracy on all five benchmarks, whereas element-level predictions from the InfoDet-trained detector bring little or no gain on the four existing benchmarks. These results suggest that the structure learned from ChartGalaxy++ benefits chart understanding.

We are also preparing QA pairs for multi-task training experiments to assess whether jointly learning scene graph prediction and chart QA improves performance on standard benchmarks. We will report the results during the discussion period if they become available.

Q3: The QA results do not show that structure matters. In the Table 3 ablation, the largest gain comes from adding text with digits, which points to OCR content rather than structure. Adding relationships gives very minor gains. A flat OCR plus boxes baseline without hierarchy is needed to isolate the contribution of the graph structure.

A3: We agree that text with numerical values brings a large gain in Table 3, while relationships contribute only about 1 point. To better distinguish the contributions of textual content and structural information, we have added the requested flat OCR + boxes baseline and an additional ablation replacing semantic node labels with neutral labels (node_i) in Appendix E.4. All settings use the same 1,266 QA questions and evaluation metric as Table 3.

SettingLabelsRelationshipsAcc. (%)
Full scene graphSemanticwith86.9
Full scene graph − relationshipsSemanticwithout85.6
Full scene graph, neutral labelsNeutralwith86.5
Full scene graph − relationships, neutral labelsNeutralwithout83.4
Flat OCR + boxesNonewithout81.8

The flat OCR + boxes baseline retains all text, including numerical values, and the corresponding bounding boxes, but excludes non-text nodes and graph relationships. As shown in the table above, the full scene graph achieves 86.9% accuracy, compared with 81.8% for this baseline.

The relationship ablations provide a more direct assessment. With semantic labels, adding relationships improves accuracy by 1.3 percentage points (85.6% to 86.9%), since the labels still convey part of the hierarchy. With neutral labels that carry no such information, the gain increases to 3.1 percentage points (83.4% to 86.5%). Semantic labels identify the entities to which elements belong and can therefore convey grouping information even when explicit relationships are removed. These results suggest that relationships benefit QA, and semantic labels partly compensate for their absence.

Moreover, the hierarchy can be further exploited through structure-aware subgraph retrieval, which raises accuracy from 86.9% to 90.1% (see our response to Q1).

References

[1] J. Zhu, et al., "InfoDet: A Dataset for Infographic Element Detection," ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer CdVW (3/3)

Q4: Venue fit. This is primarily a dataset and benchmark paper, and its contribution to representation learning is limited. Dataset papers can belong at ICLR, but in its current form this one would fit better at a venue such as the NeurIPS Datasets and Benchmarks track.

A4: We agree that the paper's primary contribution is a dataset and benchmark. Its contribution to representation learning lies in providing explicit structural supervision and evaluating how models learn, use, and preserve the hierarchical and spatial relationships in infographic charts.

  1. Learning structured infographic representations.

ChartGalaxy++ provides scene graph annotations that make hierarchical groupings and CRAP-guided spatial relationships explicit learning targets. These annotations support learning to represent not only individual visual elements but also how they are organized and related. Our image-to-scene-graph model trained on ChartGalaxy++ achieves a hierarchical F1 of 85.1, compared with 50.3 for the strongest evaluated baseline, GPT-6 Astra (Sec. 5.1). These results show that ChartGalaxy++ supports training models to recover hierarchical chart structure directly from images.

  1. Using scene graph representations for chart understanding.

Providing scene graph annotations improves QA accuracy across all ten evaluated models, with an average gain of 12.3 percentage points over direct prompting (Sec. 5.2). Moreover, scene graphs predicted by our trained model improve QA on four existing benchmarks (Appendix E.7). These findings establish the downstream utility of explicit structural information and show that the model's predicted representations are useful beyond ChartGalaxy++.

  1. Evaluating structure preservation in generation.

Scene graphs specify the elements and relationships to be rendered and allow us to evaluate whether models preserve the structure. Among eleven evaluated image-generation models, even the strongest, GPT-Image-2.5 Sunburst, achieves hierarchical and spatial F1 scores of only 75.4 and 64.2, respectively (Sec. 5.3). This evaluation identifies limitations in the evaluated models' ability to preserve hierarchical and spatial relationships, helping guide research on learning and preserving infographic structure.

Together, these experiments connect structural supervision to learned chart representations, their downstream use, and their preservation during generation. We believe this connection makes ChartGalaxy++ relevant to ICLR's representation learning community. We have revised the Introduction, Sec. 4, and Sec. 5 to make this connection and the supporting evidence more explicit.

AuthorsOpenReview ↗

Looking Forward to Further Feedback

Thank you again for your thoughtful reviews and constructive feedback. We sincerely appreciate the time and effort you have devoted to evaluating our work.

We have carefully addressed all of the concerns raised in the reviews through clarifications of our methodological contributions, additional experiments on the value of the data for training, new baselines and ablations on the role of scene graphs in QA, and a discussion of venue fit in our rebuttal. We would greatly appreciate it if you could take a moment to review our response and let us know whether they have adequately addressed your concerns.

If there are still any remaining questions or aspects that you feel require further evidence or clarification, we would be more than happy to provide additional experiments or explanations during the discussion period. Your feedback is extremely valuable in helping us improve the paper.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Author Rebuttal

Dear Reviewers, ACs, and SACs,

We sincerely thank you for your valuable and insightful comments. We are encouraged by the recognition of the value, scale, and novelty of our dataset, as well as our comprehensive evaluation and scalable annotation pipeline. Overall, the reviewers highlighted the following strengths:

The value and usefulness of the dataset: All three reviewers agreed on the value and usefulness of ChartGalaxy++.

  • R_CdVW: "The dataset is likely to be a useful resource for the community."
  • R_eE8o: "...contributing to infographic chart understanding and generation."
  • R_Uvgd: "...the dataset is directly useful for training and evaluating structure-aware chart models."

The scale and novelty of the dataset: All three reviewers recognized the large scale of ChartGalaxy++ and its novelty in annotating how visual elements are grouped and related.

  • R_CdVW: "Existing infographic datasets annotate elements in isolation, while ChartGalaxy++ explicitly records how elements group into data items, series, legends and axes, at a scale beyond prior structural chart datasets."
  • R_eE8o: "The motivation of representing infographic charts as scene graphs is clear..."
  • R_Uvgd: "Large-scale semantic hierarchy annotations for infographic charts. ChartGalaxy++ annotates 217K infographic charts with semantic groups and their hierarchy, not just individual elements. Earlier structural datasets either cover only plain charts (VisText, VisAnatomy) or group elements by layout without data semantics (Shi et al., 2023)."

Comprehensive evaluation: The reviewers appreciated our comprehensive evaluation.

  • R_eE8o: "...the three use cases of the dataset are generally convincing."
  • R_Uvgd: "...a fine-tuned Qwen3.5-4B image-to-scene-graph model that outperforms ten LVLMs..."
  • R_Uvgd: "Broad experimental coverage. The paper evaluates the dataset in three settings and compares a wide range of models: 10 LVLMs for scene graph prediction and QA, and 11 image-generation models for scene graph preservation. This gives a useful picture of where current models stand on infographic structure..."

Scalable and high-quality annotation pipeline: R_eE8o highlighted both the scalability and quality of our annotation pipeline.

  • R_eE8o: "The authors built an automated annotation pipeline with iterative refinement and complementary validation methods, scaling the annotation to >200k images with manageable cost, while showing high annotation quality in human audit. This is the most interesting part to me, and can inspire benchmark curation in other domains..."

We are also truly grateful for the constructive feedback from the reviewers and have carefully addressed their concerns. Below, we summarize the main review concerns and our responses/solutions.

Additional quantitative/qualitative analysis:

  • Conducted evaluations on existing QA benchmarks to show the utility of ChartGalaxy++ beyond its own distribution (R_CdVW, R_Uvgd).
  • Initiated multi-task training experiments on scene graph prediction and chart QA (R_CdVW).
  • Conducted comparisons with flat OCR + boxes and reconstructed SVGs, a neutral-label ablation, and significance tests to show the benefits of scene graph structure for QA (R_CdVW, R_eE8o, R_Uvgd).
  • Proposed a structure-aware subgraph retrieval method that further improves QA accuracy (R_CdVW, R_eE8o).
  • Assessed annotation quality by scoring the pre-correction pipeline outputs and auditing randomly sampled training charts (R_Uvgd).
  • Evaluated the performance of our annotation pipeline with an open model, Qwen3.8-27B (R_Uvgd).
  • Reported image-to-scene-graph results separately for real and synthetic charts (R_Uvgd).

Clarifications:

  • Clarified our contributions and their connection to representation learning (R_CdVW).
  • Clarified the experimental settings of the baselines and the QA ablation (R_eE8o, R_Uvgd).
  • Clarified the construction details of the dataset and the QA benchmark (R_Uvgd).
  • Released the complete scene graph inputs for all QA questions (R_Uvgd).
  • Discussed training image-generation models conditioned on scene graphs as future work (R_Uvgd).

We are confident that these additional analyses and clarifications will effectively address the reviewers' concerns. We deeply appreciate your careful review, constructive feedback, and valuable time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.