- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ID2C, a 1,000-example document-to-chart benchmark intended to evaluate chart generation when the relevant data, required transformation, and visualization form are left implicit. It also proposes a retrieval-augmented multi-agent pipeline that retrieves chart-form and value-transformation knowledge from a catalogue before generating and refining the visualization.
Critical strengths
-
The paper identifies a reasonable limitation of existing text-to-chart tasks.
-
The benchmark is reasonably large and carefully structured.
-
The authors conduct a human-alignment study and ablate criterion decomposition, sample-specific rubrics, and criterion-wise inference.
Critical weaknesses
-
Although the paper motivates ID2C by contrasting it with nvBench, nvBench 2.0, VisEval, MatPlotBench, Text2Vis, and DOC2CHART, the comparison is not in depth. As a result, it is difficult to assess whether ID2C actually measures a substantially new capability, is systematically more challenging, or simply repackages existing text/document-to-visualization tasks with more implicit prompts.
-
Despite proposing a chart-generation system, the experiments do not compare against existing visualization agents or document-to-chart systems. The primary system evaluation is an ablation among the authors' own single-shot, multi-agent, and retrieval-augmented variants. Therefore, the reported gains establish that retrieval helps their particular pipeline, but do not establish that the resulting system advances the state of the art.
-
The proposed harness largely combines well-established components.
Questions for authors
See weaknesses
3 comments
(W2) Comparison with existing visualization systems
We agree that comparisons with existing visualization systems are needed to show the effectiveness of the proposed harness. We have added evaluations of Doc2Chart [1] and MatPlotAgent [2], adapting both methods to our document-to-chart setting and using the same Qwen3.6-27B backbone across all systems.
| System | Task fulfillment | Data correctness | Readability | Overall |
|---|---|---|---|---|
| Doc2Chart | 61.3 | 71.7 | 63.0 | 66.0 |
| MatPlotAgent | 63.0 | 78.9 | 71.0 | 71.0 |
| Single-shot | 46.3 | 75.4 | 60.0 | 60.7 |
| Multi-agent pipeline | 60.9 | 77.5 | 76.5 | 70.2 |
| Catalogue retrieval (Ours) | 70.5 | 77.2 | 70.2 | 73.3 |
Our harness outperforms both existing systems overall, with the clearest advantage in task fulfillment: it exceeds the stronger baseline, MatPlotAgent, by 7.5 points. This supports its effectiveness in addressing the analytical-planning difficulties identified by ID2C. We will include these comparisons in the revised paper.
[1] Akriti Jain, Pritika Ramu, Aparna Garimella, and Apoorv Saxena. “Doc2Chart: Intent-Driven Zero-Shot Chart Generation from Documents.” EMNLP, 2025.
[2] Zhiyu Yang et al. “MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization.” Findings of ACL, 2024.
(W3) Contribution of the proposed harness
We agree that the proposed harness can be viewed as a combination of well-established techniques. The primary contribution of this work is the problem formulation and benchmark. Our analysis in Section 5.2 identifies inferring appropriate value transformations and chart forms as major difficulties under implicit requests. Guided by this finding, we proposed a task-specific adaptation of these techniques, particularly by equipping the form-planning agent with retrieval from a catalogue of value transformations and chart forms. This design directly addresses the planning difficulties identified by our analysis, and our controlled experiments show improvements concentrated on the corresponding evaluation criteria. We will clarify this positioning in the revised paper.
Thanks for your responses. I will raise my score
We thank the reviewer for the feedback on benchmark differentiation, comparisons with existing systems, and the contribution of the proposed harness.
(W1) Comparison with prior benchmarks
We agree that the related-work discussion should explain the distinction at the level of inputs, requested decisions, and evaluation protocols. ID2C focuses on the combination of document-grounded evidence, implicit analytical and visual requirements, and evaluation that accommodates multiple chart forms.
To make the comparison more concrete, we provide representative requests from the benchmarks discussed in the paper, alongside an ID2C example:
ID2C explicitly evaluates whether a system infers and fulfills unstated analytical and visual requirements for a given document needed to answer the request. This capability is not separately assessed by prior works, including DOC2CHART, whose primary automated metrics focus on data attribution and reference chart-type agreement.
ID2C also extends the evaluation scope from financial reports and scientific papers to heterogeneous real-world documents across domains. Together, its document coverage and requirement-specific criteria enable systematic diagnosis of analytical-planning failures. It also provides a public resource for evaluating document-grounded chart generation, addressing a practical gap left by DOC2CHART’s unreleased dataset.
These contributions go beyond rephrasing existing visualization requests. We will clarify these distinctions in the revised paper.