- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VisAudit, a benchmark for evaluating the ability of multimodal agents to autonomously diagnose, repair, and verify data visualizations. The benchmark comprises 1,900 flawed chart instances spanning 21 chart types and 10 flaw categories, along with 300 initially correct charts. The authors design three evaluation tracks covering diagnosed repair, autonomous repair, and open-world verification, and assess repair effectiveness using metrics such as recovery and preservation. Through experiments on seven multimodal models, the authors identify significant limitations in autonomous defect diagnosis, multi-flaw repair, and avoiding unnecessary modifications to already-correct charts.
Critical strengths
-
Practical Relevance. The paper addresses a meaningful research problem by focusing on the complete workflow of autonomous defect detection, repair, and verification, closely reflecting how agents would review and correct visualizations in real-world scenarios.
-
Comprehensive and Well-Designed Benchmark. The proposed benchmark provides a comprehensive evaluation of different visualization defects and introduces three progressively challenging evaluation tracks covering diagnosed repair, autonomous diagnosis and repair, and open-world verification. This progressive design enables a systematic assessment of agents' capabilities at different stages of the visualization review process.
Critical weaknesses
-
Dataset Realism and Generalization Remain Unverified. The paper primarily synthesizes flawed charts using predefined perturbations. However, it remains unclear whether these artificially introduced defects adequately reflect the complexity and diversity of visualization errors encountered in real-world scenarios. Further justification and validation are needed.
-
Insufficient Human Evaluation. The authors primarily rely on structured chart attribute matching to evaluate repair quality. However, it has not been sufficiently demonstrated that these automated metrics align with human judgments of actual visual repair quality. Additional human evaluations are recommended to validate the reliability of the proposed metrics.
-
Limited Qualitative Visual Examples. The paper lacks sufficient qualitative examples illustrating different types of chart defects and their corresponding before-and-after repair results. Providing more visual comparisons across different error categories would help readers better understand the dataset composition and evaluation methodology.
-
Lack of Fine-Grained Failure Case Analysis. Although the paper presents quantitative results across different defect categories and numbers of defects, it provides limited analysis of specific failure cases and the agent's diagnosis, repair, and verification processes. Including visualizations of representative failure trajectories would help clarify the underlying reasons for the agent's failures.
Questions for authors
Same as Weakness.
3 comments
Weakness 3: Limited Qualitative Visual Examples. The paper lacks sufficient qualitative examples illustrating different types of chart defects and their corresponding before-and-after repair results. Providing more visual comparisons across different error categories would help readers better understand the dataset composition and evaluation methodology.
Answer: We have added a new qualitative analysis in Appendix C.1 of the newly uploaded paper, presenting representative before-and-after visualizations across three defect categories: color contradiction, dropped series, and wrong chart type. Each example includes the ground-truth visualization, the flawed input, and the agent's final repair. These examples illustrate how different types of visualization defects manifest and how agents successfully correct them while preserving the intended content. They also demonstrate that semantically correct repairs need not exactly reproduce the reference visualization. We believe these additional examples provide a clearer understanding of the benchmark's defect categories and evaluation methodology.
Weakness 4: Lack of Fine-Grained Failure Case Analysis. Although the paper presents quantitative results across different defect categories and numbers of defects, it provides limited analysis of specific failure cases and the agent's diagnosis, repair, and verification processes. Including visualizations of representative failure trajectories would help clarify the underlying reasons for the agent's failures.
Answer: To provide a more fine-grained understanding of agent failures, we have added Appendix C.2 in the newly uploaded paper, which presents three representative failure trajectories illustrating distinct challenges in the diagnosis–repair–verification process. Specifically, the examples demonstrate: (1) an incomplete diagnosis that leads to partial repair, where the agent restores a missing series but overlooks axis truncation; (2) a regression during iterative repair, where the agent initially produces a correct visualization but subsequently removes a necessary legend; and (3) a verification failure, where the agent corrects a visible color inconsistency but fails to identify an incorrect numerical reconstruction. For each case, we present the original visualization, flawed input, and repair outcome, together with an analysis of the agent's intermediate actions and verification behavior. These examples reveal how failures can arise from incomplete diagnosis, unintended modifications, and insufficient verification, providing qualitative insights that complement our quantitative evaluation.
[D1] Lo, Leo Yu‐Ho, et al. "Misinformed by visualization: What do we learn from misinformative visualizations?", Computer Graphics Forum, 2022.
[D2] Tonglet, Jonathan, et al. "Is this chart lying to me? Automating the detection of misleading visualizations." Annual Meeting of the Association for Computational Linguistics, 2026.
Response to author
Hi, apologies for my late response.
Thank you for the additional experiments and clarifications. While they address some of my concerns, I still have reservations about the overall persuasiveness of the experimental evidence.
Thank you for your constructive comments. We believe that our discussion will lead to a better paper in general. We address your comments in a point-by-point manner.
Weakness 1: Dataset Realism and Generalization Remain Unverified. The paper primarily synthesizes flawed charts using predefined perturbations. However, it remains unclear whether these artificially introduced defects adequately reflect the complexity and diversity of visualization errors encountered in real-world scenarios. Further justification and validation are needed.
Answer: The ten flaw categories considered in VisAudit are grounded in visualization errors documented in prior studies of real-world charts. In particular, [D1] analyzed over 1,000 real-world misleading visualizations and identified 74 types of visualization issues, including incorrect or misleading labels, missing legends, plotting errors, inappropriate chart types, problematic axis ranges, and misleading visual encodings, which closely correspond to the flaw categories considered in VisAudit. Controlled synthetic generation is also adopted in existing visualization benchmarks, such as Misviz [D2], which includes synthetically generated misleading charts alongside real-world examples. Our controlled perturbations are therefore intended to instantiate documented real-world classes of visualization errors while enabling systematic evaluation with precisely identified defects and reference properties.
Weakness 2: Insufficient Human Evaluation. The authors primarily rely on structured chart attribute matching to evaluate repair quality. However, it has not been sufficiently demonstrated that these automated metrics align with human judgments of actual visual repair quality. Additional human evaluations are recommended to validate the reliability of the proposed metrics.
Answer: To assess whether our automatic evaluation metrics align with human judgments of repair quality, we conducted an additional human evaluation on 200 outputs generated by Claude Haiku. The samples were selected through stratified sampling to match the benchmark's distribution of single-flaw, double-flaw, and triple-flaw instances. Unlike our original human validation of generated benchmark tasks, this evaluation directly examines actual agent-generated repairs. Human annotators independently assessed whether each repaired visualization successfully corrected all original defects (recovery) and preserved all unaffected chart components (preservation). Importantly, human judgments were based on semantic correctness, allowing valid alternative repairs that do not exactly reproduce the reference visualization.
The table below compares the number of successful instances identified by human annotators and our automatic evaluation pipeline, together with their instance-level agreement rates and false-positive/false-negative rates.
The results show substantial agreement between human judgments and our automatic evaluation pipeline. The relatively low false-positive rates indicate that the pipeline rarely identifies unsuccessful repairs as successful. However, the automatic evaluation is more conservative than human judgment, particularly for recovery, where the automatic success rate is 31.5% compared with 44.0% under human evaluation, with a false-negative rate of 33.0%. This discrepancy suggests that some semantically correct repairs are not fully captured by structural property matching. Nevertheless, the high overall agreement supports the reliability of our automatic metrics, while highlighting their tendency to underestimate successful recovery rather than overestimate agent performance.