- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ChartLogic, a Parse-and-Connect framework that maps chart images to ontology-typed instance graphs (ABoxes). The Parse stage recovers visual instances and hierarchical identities, while the Connect stage predicts relations such as axis membership and legend–mark correspondence. The recovered graph is evaluated through structural SPARQL queries rather than only answer- or table-level metrics. The authors train on a synthetic dataset with aligned images, instance masks, and ABoxes, and evaluate on both a held-out synthetic set and a re-rendered ChartQA-derived subset. The experiments focus on series identity recovery, open-recovery versus ground-truth-skeleton evaluation, and attribution of failures to parsing or relation binding.
Critical strengths
-
Fine-grained structural evaluation. The separation of parsing and relational binding, especially the comparison between open recovery and ground-truth-skeleton evaluation, provides a useful diagnostic framework for distinguishing perception failures from relation-prediction failures. This is a meaningful perspective beyond answer-level chart evaluation.
-
Well-aligned synthetic supervision. The shared generation pipeline provides exact alignment among chart images, instance-level masks, and symbolic ABox annotations. This makes it possible to study instance identity and structural bindings under controlled conditions without requiring expensive manual graph annotations.
Critical weaknesses
-
The proposed structural task is not sufficiently validated as a better measure of chart understanding.
The paper motivates instance-graph recovery by arguing that answer- or table-level metrics may hide structural errors, but it does not demonstrate that the proposed task is in fact a better proxy for chart understanding. The evaluation is limited to structural queries defined within the proposed ABox representation, such as axis membership, series grouping, and legend–mark correspondence, without showing whether better structural recovery predicts or improves performance on downstream tasks such as ChartQA. The paper also does not evaluate a representative set of existing chart models to show that conventional benchmarks fail to distinguish models while the proposed task reveals meaningful differences. As a result, the experiments show that the proposed metrics can detect structural errors, but not that these errors are consequential for broader chart understanding. -
The method contribution of ChartLogic is difficult to assess due to the lack of competitive baselines and limited external validation.
ChartLogic is evaluated mainly through internal ablations and comparisons against heuristic or ground-truth-skeleton variants, rather than against representative prior chart parsing, derendering, or structured scene-graph methods. This makes it difficult to determine whether ChartLogic itself provides an advantage over existing approaches or whether the reported performance mainly reflects the proposed supervision and evaluation setup. In addition, the main experiments use charts generated by the authors’ own renderer, while verify-120 is also re-rendered through the same pipeline and is evaluated only at the Parse level rather than with full open recovery. Therefore, both the comparative effectiveness and real-world generalization of the proposed method remain unclear.
Questions for authors
Clarification requests
-
What do the authors consider the primary contribution of the paper: a new chart-understanding task/evaluation protocol, a new dataset/benchmark, or the ChartLogic model itself? The current presentation appears to combine all three, and clarifying this would affect how I assess the required empirical validation.
-
Are there existing chart parsing, derendering, scene-graph, or multimodal models that can be adapted to the proposed ABox recovery task? If so, could the authors clarify why no such external model baselines are included in the evaluation?
Assessment-critical empirical questions
-
Can the authors provide evidence that the proposed structural-recovery task captures chart understanding beyond what existing answer- or table-level benchmarks measure? For example, do different existing chart models exhibit similar ChartQA/table-recovery performance but substantially different structural-recovery performance, and are these differences associated with meaningful downstream failures?
-
Is there any evidence that better ABox recovery translates into better downstream chart reasoning, such as ChartQA performance? The current evaluation demonstrates that the proposed metrics detect structural errors, but it is unclear whether these errors are consequential for downstream chart understanding.
-
Do the authors have results for full open recovery on charts outside their own rendering pipeline, rather than only on synthetic charts and the re-rendered verify-120 subset? Such evidence would materially affect my assessment of the external validity of the proposed task and method.
Response to Cngv: what the paper claims, which baselines we ran, and whether the errors matter
Dear Reviewer Cngv,
Thank you — your review is the one that forced us to say what the paper claims. You asked for one contribution, for a representative set of existing models, and for evidence that structural errors matter; we answer in that order.
Q1 — Task, benchmark, or model?
One contribution: a measurement protocol, with the model as its instrument. The protocol scores a recovered instance graph so that misses, wrong bindings and unsupported assertions are separated and priced differently. Where the abstract now reads as three co-equal contributions, the revision will state one contribution and one set of comparisons (Abstract and §5).
Q2 — A representative set of existing models
Four families, run on the same inputs. Two can be placed on our predicates, two cannot:
The last column mixes two protocols, and we do not compare across it: ChartOCR and our model are scored on mask-aligned binding, the MLLMs on a semantic probe that credits a mapping stated in text without anchoring it to a handle.
LineFormer and LineEX emit masks or keypoints without a binding layer; structured scene-graph models such as RelTR have no chart checkpoint, and re-training one on charts would replace its supervision with ours. Thus the absence you note is partly a property of the field: most chart systems do not produce a graph on which binding can be scored. This table goes into §4.4.
Q3 / W1 — Is the structural task a better measure?
At the metric level and, in a controlled setting, at the consequence level; not at the model level. Same 400 charts, same predictions:
Thus a table-level score cannot tell a chart type that is nearly solved from one that is not. For consequence, we corrupt the reference graph and hold the tuple total fixed: damaging 5% of belongsToAxis drops axis queries from 1.000 to 0.445 while counting stays at 1.000; damaging 5% of usesDataStructure drops counting from 1.000 to 0.380. Structural error is not uniform, and counting the wrong tuples hides which query breaks.
Q4 — Downstream ChartQA
In part, on a controlled version of the question. We answer 646 templated questions with a deterministic program reasoner — no QA head — from the gold calibrated table and ABox of verify-120, and from our predicted recovery. Accuracy is 0.580 against 1.000 from gold: a drop of 0.420, with 0.110 unanswerable against none. The gold arm is 1 by construction, so it measures no difficulty; what it measures is what recovery error costs, and it costs two answers in five. Nor is it the tuple error restated: 0.374 of reference tuples are unmatched, yet answers fall by 0.420. Thus within this probe the errors are consequential. It remains a probe, not a ChartQA result: templates are not native ChartQA language, the reasoner is a program not a trained head, and scoring is exact or tolerance. The revision will report this at §4.5 and keep a trained QA head as a limitation at §5.
Q5 / W2 — The supervision and evaluation setup as the main factor
In part, and the comparison least exposed to it can be named. Every synthetic result uses our generator and supervision; shuffled versus real series supervision moves Acccs from 0.039 to 0.959, more than any architectural effect, so we no longer present the model as an architecture contribution. The comparison that does not rest on our annotations is the CHART-Info set [1]: image only, organizers' script, 396 charts, no system sees the scored ground truth. Extraction there is level (−0.4). Thus our claim is about what the protocol reveals, not about beating a parser on its own task. The revision will report both at §4.4 and §4.6, and state one contribution instead of claiming a model advantage.