- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ChartGalaxy++, a dataset of 217,195 infographic charts built on ChartGalaxy, each annotated with a scene graph that groups visual elements into semantic groups encoding their hierarchical and spatial relationships. Real charts are annotated by an OCR/detection pipeline combined with Gemini 3.8 Flash for attributes and grouping, while synthetic ones are parsed from their source SVGs, and the annotations of a 1,000-image test set are manually verified. The authors validate the dataset through three applications: (1) a fine-tuned Qwen3.5-4B image-to-scene-graph model that outperforms ten LVLMs; (2) a QA benchmark where adding annotated scene graphs to the prompt improves accuracy by 8.9 points on average; and (3) an evaluation of scene graph preservation in 11 image-generation models.
Critical strengths
1. Large-scale semantic hierarchy annotations for infographic charts. ChartGalaxy++ annotates 217K infographic charts with semantic groups and their hierarchy, not just individual elements. Earlier structural datasets either cover only plain charts (VisText, VisAnatomy) or group elements by layout without data semantics (Shi et al., 2023). This is the main reason I lean toward acceptance, since the dataset is directly useful for training and evaluating structure-aware chart models.
2. Broad experimental coverage. The paper evaluates the dataset in three settings and compares a wide range of models: 10 LVLMs for scene graph prediction and QA, and 11 image-generation models for scene graph preservation. This gives a useful picture of where current models stand on infographic structure, for example that even GPT-6 Astra reaches only 50.3 hierarchical F1.
Critical weaknesses
1. The experiments do not show that the dataset improves models beyond its own distribution. The image-to-scene-graph comparison in Sec. 5.1 pits a model fine-tuned on ChartGalaxy++ against zero-shot LVLMs on an in-distribution test set annotated under the same conventions. Outperforming them is expected and does not by itself show the value of the dataset. The QA results in Sec. 5.2 rely on scene graphs produced by the annotation pipeline rather than predicted by the fine-tuned model, and are reported only on the authors' own 1,266-question benchmark. Table 3 further suggests that most of the QA gain comes from node text with digits rather than relationships, and the concise node labels may already state which entity a value belongs to, which would turn within-group questions into a lookup. This matters because these two applications are the main evidence that the dataset helps chart understanding. The authors should evaluate the fine-tuned model on out-of-distribution infographics, report QA results using its predicted scene graphs on existing benchmarks such as InfoChartQA, InfographicVQA, or ChartQAPro, and include a flat OCR-plus-box baseline and models trained on alternative data (e.g., InfoDet or only the real or synthetic subset).
2. The annotations come from a single model, and their reliability is not well established. For real charts, element types, descriptions, and the entire hierarchy are produced and verified by Gemini 3.8 Flash, and the training set receives no human verification. The audit covers only a curated test set and corrects pipeline outputs rather than annotating independently, so the reported error rates likely overestimate overall quality. The grouping also follows detailed conventions (Prompt B.6), and it is unclear whether the baselines in Sec. 5.1 are given them. If not, they are scored against conventions they never saw, and since hierarchical F1 only credits exact direct parents, a single extra or missing group level makes all edges below it count as errors. This matters because every result in the paper rests on these annotations. The authors should audit a random training sample, report the pipeline's own score on the test set before correction, and evaluate baselines with the grouping rules provided.
Questions for authors
Clarifications
- Are the baselines in Sec. 5.1 given the grouping rules in Prompt B.6, or only the scene graph schema? If only the schema, how much of the hierarchical F1 gap remains when the rules are provided?
- Do the concise QA labels (Appendix E.3) name the entity a value belongs to? If so, the association is available from the label text alone, and the QA gain cannot be attributed to the hierarchy. Please provide a complete scene graph input for one QA question.
- In Table 3, does "relationships" include parent references or only spatial relations? If it includes parents, "Best − relationships" (85.6) is effectively a flat baseline.
- Of the 1,000 charts in the QA benchmark, how many belong to the human-verified test set, and how many are real versus synthetic?
- How are SVG
<g>groups in synthetic charts mapped to the seven semantic group types, and are the resulting hierarchies consistent with the conventions in Prompt B.6?
Additional results that could change my assessment
- QA accuracy with scene graphs predicted by the fine-tuned model, ideally also on an existing benchmark such as InfoChartQA, InfographicVQA, or ChartQAPro.
- QA accuracy with neutral node labels (e.g.,
text_12) and, if not covered by Table 3, with a flat OCR-plus-box input. - The pre-correction pipeline output on the test set scored with the Table 1 metrics, and error rates from an audit of a random training sample.
- Table 1 results split by real and synthetic test charts.
Suggestions for future work (not affecting my score)
- Training an image-generation model conditioned on scene graphs, rather than only evaluating existing models.
- Replacing Gemini with an open model in the annotation pipeline, to assess how much the dataset depends on a single proprietary model.
7 comments
Response to Reviewer Uvgd (2/5)
Q1 (continued)
A1 (continued): 3. Contributions of semantic labels and explicit relationships to QA performance.
We agree that concise node labels can convey entity associations and make some within-group questions easier to answer through lookup.
For example, the shared entity name in value_label_land_bank_of_the_philippines and data_item_land_bank_of_the_philippines suggests their association.
However, the scene graph structure makes this association explicit through its relationships.
To separate these contributions, we have added an ablation that replaces semantic labels with neutral labels (node_i), both with and without relationships (Appendix E.4).
| Setting | Labels | Relationships | Acc. (%) | vs. Full scene graph |
|---|---|---|---|---|
| Full scene graph | Semantic | with | 86.9 | — |
| Full scene graph − relationships | Semantic | without | 85.6 | −1.3 |
| Full scene graph, neutral labels | Neutral | with | 86.5 | −0.4 |
| Full scene graph − relationships, neutral labels | Neutral | without | 83.4 | −3.5 |
With semantic labels, adding relationships improves accuracy by 1.3 percentage points (85.6% to 86.9%), since the labels still convey part of the hierarchy. With neutral labels that carry no such information, the gain increases to 3.1 percentage points (83.4% to 86.5%). Semantic labels identify the entities to which elements belong and can therefore convey grouping information even when explicit relationships are removed. These results suggest that relationships benefit QA, and semantic labels partly compensate for their absence.
Q2: The annotations come from a single model, and their reliability is not well established.
A2: Per your suggestion, we have evaluated the pre-correction pipeline outputs on the test set with the Table 1 metrics and manually audited 500 randomly sampled training charts. We also clarify that the baselines in Sec. 5.1 are already provided with the grouping rules.
- Pre-correction pipeline outputs on the test set.
Our annotation-quality assessment in Appendix B.3 records the errors in the pipeline outputs before correction. We further score these pre-correction outputs against the human-corrected test set annotations with the Table 1 metrics. All scores below are on a 0–100 scale.
| Test subset | Size | Overall | Type | Box | Text | Color | Description | Hierarchical | Spatial |
|---|---|---|---|---|---|---|---|---|---|
| All | 1,000 | 99.16 | 99.95 | 99.75 | 99.70 | 99.81 | 99.57 | 99.30 | 98.75 |
| Real | 500 | 98.79 | 99.91 | 99.52 | 99.60 | 99.81 | 99.55 | 99.24 | 98.15 |
| Synthetic | 500 | 99.52 | 99.98 | 99.91 | 99.79 | 99.81 | 99.58 | 99.45 | 99.35 |
As shown in the table above, the pre-correction pipeline outputs achieve an overall node-level F1 of 99.16 and a hierarchical F1 of 99.30, with similar hierarchical scores for real and synthetic charts (99.24 and 99.45).
- Audit of randomly sampled training charts.
We randomly sample 500 charts from the 216,195 training charts and manually assess their annotations following the protocol in Appendix B.3.
| Metric | Existing test-set assessment (1,000 charts) | Random training-sample audit (500 charts) |
|---|---|---|
| Element precision | 99.81% | 99.80% |
| Element recall | 99.78% | 97.75% |
| Mean box IoU | 0.9976 | 0.9929 |
| Text transcription error | 0.39% (134/34,242) | 1.29% (171/13,256) |
| Numeric transcription error | 0.12% (27/21,856) | 0.12% (10/8,485) |
| Description / role error | 3.08% (3,053/99,010) | 5.39% (2,312/42,896) |
| Incorrect parent assignment | 0.010% (10/99,010) | 0.040% (17/42,896) |
The training sample retains high element precision and box IoU, with the same numeric transcription error rate as the test set (0.12%). Incorrect parent assignments remain rare (0.040%). However, element recall is slightly lower (97.75% vs. 99.78%). Text transcription errors increase from 0.39% to 1.29%, and description/role errors from 3.08% to 5.39%.
- Grouping rules provided to the baselines.
We would like to clarify that the baselines in Sec. 5.1 are provided with both the scene graph schema and the grouping rules in Prompt B.6. We have revised Sec. 5.1 to state this explicitly, and added the baseline prompt to Appendix D.2.
Q3: Are the baselines in Sec. 5.1 given the grouping rules in Prompt B.6, or only the scene graph schema? If only the schema, how much of the hierarchical F1 gap remains when the rules are provided?
A3: We would like to clarify that the baselines are provided with both the scene graph schema and the grouping rules in Prompt B.6. We have revised Sec. 5.1 to state this explicitly, and added the baseline prompt to Appendix D.2.
Response to Reviewer Uvgd (3/5)
Q4: Do the concise QA labels (Appendix E.3) name the entity a value belongs to? If so, the association is available from the label text alone, and the QA gain cannot be attributed to the hierarchy. Please provide a complete scene graph input for one QA question.
A4: Yes, as described in Appendix E.3, concise node labels summarize node descriptions and grouping context, and can therefore identify the entity to which a value belongs. We agree that the QA gain cannot be attributed solely to the hierarchy. To separate the contributions of semantic labels and explicit relationships, we added a neutral-label ablation (Appendix E.4; see Q1). With neutral labels, accuracy remains 86.5%, compared with 86.9% with semantic labels. Removing relationships under the same neutral-label setting reduces accuracy to 83.4%. This 3.1-point difference supports a contribution from both hierarchical and spatial relationships.
Due to the response length limit, we present selected parts of the scene graph JSON inputs for the four examples in Appendix E.1. The complete scene graph inputs for all QA questions are available in our GitHub repository (https://github.com/ChartGalaxyPP/ChartGalaxyPlusPlus/releases/download/v1.0/qa.tar.gz).
Q5: In Table 3, does "relationships" include parent references or only spatial relations? If it includes parents, "Best − relationships" (85.6) is effectively a flat baseline.
A5: In Table 3, "relationships" includes both hierarchical parent references and spatial relations. Although "Best − relationships" (85.6%) is structurally flat, it is not equivalent to a flat baseline with neutral labels, because its semantic labels still implicitly encode part of the hierarchy. To separate these sources of information, we added a flat baseline with neutral labels (node_i), retaining node types, bounding boxes, and text (Appendix E.4; see also our response to Q1). This baseline achieves 83.4% accuracy. Adding relationships while keeping neutral labels raises accuracy to 86.5% (+3.1 points); restoring both relationships and semantic labels yields 86.9% (+3.5 points).
Q6: Of the 1,000 charts in the QA benchmark, how many belong to the human-verified test set, and how many are real versus synthetic?
A6: The QA benchmark contains 1,000 charts: 500 real and 500 synthetic. Of these, 145 (14.5%) also belong to the human-verified test set, including 66 real and 79 synthetic charts.
Q7: How are SVG
<g>groups in synthetic charts mapped to the seven semantic group types, and are the resulting hierarchies consistent with the conventions in Prompt B.6?
A7:
We modify the chart-generation code to explicitly label each SVG <g> group with one of the seven semantic group types.
The renderer records these labels and the group nesting in the SVG, from which we construct the scene graph.
The resulting hierarchies follow the same annotation conventions as those used for real charts in Prompt B.6.
Q8: QA accuracy with scene graphs predicted by the fine-tuned model, ideally also on an existing benchmark such as InfoChartQA, InfographicVQA, or ChartQAPro.
A8: We evaluated QA using scene graphs predicted by our fine-tuned model on our benchmark and four existing benchmarks: InfoChartQA, InfographicVQA, ChartQAPro, and CharXiv. These predicted scene graphs yield the highest QA scores among the evaluated settings on all five benchmarks. Please refer to our response to Q1 for the detailed results and baseline comparisons.
Response to Reviewer Uvgd (1/5)
We sincerely thank you for your detailed review and thoughtful suggestions. We have carefully addressed the concerns regarding the generalization of ChartGalaxy++ beyond its own distribution, the reliability of the annotations, the settings of the baselines and the QA ablation, the construction of scene graphs for synthetic charts, the training of image-generation models, and the use of an open model in our annotation pipeline.
A1: Per your suggestions, we have added evaluations of scene graphs predicted by our fine-tuned model on out-of-distribution infographics and assessed their utility for QA on four existing benchmarks through comparisons with additional baselines (Appendix E.7). We have also added an ablation that separates the contributions of semantic labels and explicit relationships (Appendix E.4).
We randomly sample 300 charts from three existing benchmarks, 100 each from InfoChartQA, InfographicVQA, and ChartQAPro, and manually verify the scene graphs predicted by our fine-tuned model following the assessment protocol in Appendix B.3, covering 33,271 non-root nodes.
The predicted scene graphs recover most elements on OOD charts, with element precision and recall above 91% and a mean box IoU of 0.9264. Text and numeric transcription errors remain below 2% (1.53% and 1.48%), and the description or role error rate is 5.31%. These results provide evidence that the model can extract chart elements beyond ChartGalaxy++.
We evaluate Qwen3.8-27B's QA performance on four existing benchmarks, InfoChartQA, InfographicVQA, ChartQAPro, and CharXiv, as well as on our benchmark (Appendix E.7). We compare four settings: (1) direct prompting, (2) flat OCR and bounding boxes, (3) elements predicted by the detector released with InfoDet, and (4) scene graphs predicted by our fine-tuned image-to-scene-graph model. All settings use Qwen3.8-27B as the QA model. The scene graphs in this experiment are model predictions, rather than outputs of the annotation pipeline.
We report ANLS on InfographicVQA and accuracy on the other benchmarks.
Among the evaluated settings, predicted scene graphs achieve the highest QA accuracy on all five benchmarks. On our benchmark, they improve accuracy by 8.7 percentage points over direct prompting and by 2.0 points over flat OCR + boxes. On the four external benchmarks, they improve accuracy over direct prompting by 0.6, 0.6, 0.5, and 0.8 points, respectively. In comparison, the flat OCR + boxes input and the InfoDet-trained detector yield gains of at most +0.3 and +0.2 points. These external gains are modest, but they provide evidence that the utility of predicted scene graphs extends beyond our own benchmark. Their smaller magnitude may reflect the broader mix of questions in these benchmarks, whereas ours focuses on questions requiring element associations. We also observe larger gains over direct prompting on question types that depend on such associations, such as ranking in InfoChartQA (+1.7) and counting in InfographicVQA (+1.5) (Appendix E.7). We are also training the image-to-scene-graph model on the real and synthetic subsets separately. We will report the results during the discussion period if they become available.