acceptodds
Under review as a conference paper at ICLR 2027

VisAudit: Evaluating Multimodal Agents for Visual Diagnosis and Repair

Abstract

Multimodal agents are increasingly applied to data visualization tasks, but remain limited in autonomous review: unlike human reviewers, they may fail to recognize when a visualization is incorrect, determine what should be changed, repair it without disrupting correct content, and verify whether the intervention succeeded. Existing benchmarks largely evaluate predefined individual capabilities such as chart generation, instruction-guided editing, or defect detection, and therefore do not capture this gap in autonomous review. We introduce VisAudit, a benchmark for evaluating the diagnosis, repair, and verification of autonomous visualization. Given a rendered chart and configurable auxiliary evidence, including its source data table, intended text summary, and visualization code, an agent iteratively diagnoses potential defects, modifies and executes visualization code, inspects execution and visual feedback, and determines when no further intervention is needed. VisAudit defines three tracks spanning diagnosed repair, autonomous repair, and open-world verification, and contains 1,900 flawed instances across 21 chart types and 10 flaw categories, together with 300 initially correct charts. We construct the benchmark through controlled perturbations of validated source visualizations, with systematic verification and human-aligned quality control to ensure that injected defects are well-defined and recoverable from the available evidence. Experiments with leading multimodal models reveal a substantial gap from reliable autonomous review: the strongest evaluated model fully recovers only 47.4% of flawed charts in the autonomous-repair setting. By exposing failures across diagnosis, intervention, preservation, and verification, VisAudit characterizes the capabilities needed to close the gap between current agents and human-like visualization review and provides actionable insights for designing more reliable autonomous multimodal systems.

Primary area:
datasets and benchmarks
Keywords:
agentic benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ammYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VisAudit, a benchmark for evaluating the ability of multimodal agents to autonomously diagnose, repair, and verify data visualizations. The benchmark comprises 1,900 flawed chart instances spanning 21 chart types and 10 flaw categories, along with 300 initially correct charts. The authors design three evaluation tracks covering diagnosed repair, autonomous repair, and open-world verification, and assess repair effectiveness using metrics such as recovery and preservation. Through experiments on seven multimodal models, the authors identify significant limitations in autonomous defect diagnosis, multi-flaw repair, and avoiding unnecessary modifications to already-correct charts.

Critical strengths
  1. Practical Relevance. The paper addresses a meaningful research problem by focusing on the complete workflow of autonomous defect detection, repair, and verification, closely reflecting how agents would review and correct visualizations in real-world scenarios.

  2. Comprehensive and Well-Designed Benchmark. The proposed benchmark provides a comprehensive evaluation of different visualization defects and introduces three progressively challenging evaluation tracks covering diagnosed repair, autonomous diagnosis and repair, and open-world verification. This progressive design enables a systematic assessment of agents' capabilities at different stages of the visualization review process.

Critical weaknesses
  1. Dataset Realism and Generalization Remain Unverified. The paper primarily synthesizes flawed charts using predefined perturbations. However, it remains unclear whether these artificially introduced defects adequately reflect the complexity and diversity of visualization errors encountered in real-world scenarios. Further justification and validation are needed.

  2. Insufficient Human Evaluation. The authors primarily rely on structured chart attribute matching to evaluate repair quality. However, it has not been sufficiently demonstrated that these automated metrics align with human judgments of actual visual repair quality. Additional human evaluations are recommended to validate the reliability of the proposed metrics.

  3. Limited Qualitative Visual Examples. The paper lacks sufficient qualitative examples illustrating different types of chart defects and their corresponding before-and-after repair results. Providing more visual comparisons across different error categories would help readers better understand the dataset composition and evaluation methodology.

  4. Lack of Fine-Grained Failure Case Analysis. Although the paper presents quantitative results across different defect categories and numbers of defects, it provides limited analysis of specific failure cases and the agent's diagnosis, repair, and verification processes. Including visualizations of representative failure trajectories would help clarify the underlying reasons for the agent's failures.

Questions for authors

Same as Weakness.

3 comments

Thank you for your constructive comments. We believe that our discussion will lead to a better paper in general. We address your comments in a point-by-point manner.

Weakness 1: Dataset Realism and Generalization Remain Unverified. The paper primarily synthesizes flawed charts using predefined perturbations. However, it remains unclear whether these artificially introduced defects adequately reflect the complexity and diversity of visualization errors encountered in real-world scenarios. Further justification and validation are needed.
Answer: The ten flaw categories considered in VisAudit are grounded in visualization errors documented in prior studies of real-world charts. In particular, [D1] analyzed over 1,000 real-world misleading visualizations and identified 74 types of visualization issues, including incorrect or misleading labels, missing legends, plotting errors, inappropriate chart types, problematic axis ranges, and misleading visual encodings, which closely correspond to the flaw categories considered in VisAudit. Controlled synthetic generation is also adopted in existing visualization benchmarks, such as Misviz [D2], which includes synthetically generated misleading charts alongside real-world examples. Our controlled perturbations are therefore intended to instantiate documented real-world classes of visualization errors while enabling systematic evaluation with precisely identified defects and reference properties.

Weakness 2: Insufficient Human Evaluation. The authors primarily rely on structured chart attribute matching to evaluate repair quality. However, it has not been sufficiently demonstrated that these automated metrics align with human judgments of actual visual repair quality. Additional human evaluations are recommended to validate the reliability of the proposed metrics.
Answer: To assess whether our automatic evaluation metrics align with human judgments of repair quality, we conducted an additional human evaluation on 200 outputs generated by Claude Haiku. The samples were selected through stratified sampling to match the benchmark's distribution of single-flaw, double-flaw, and triple-flaw instances. Unlike our original human validation of generated benchmark tasks, this evaluation directly examines actual agent-generated repairs. Human annotators independently assessed whether each repaired visualization successfully corrected all original defects (recovery) and preserved all unaffected chart components (preservation). Importantly, human judgments were based on semantic correctness, allowing valid alternative repairs that do not exactly reproduce the reference visualization.
The table below compares the number of successful instances identified by human annotators and our automatic evaluation pipeline, together with their instance-level agreement rates and false-positive/false-negative rates.

MetricHumanAutomaticAgreement (%)FPR (%)FNR (%)
Recovery (Rec-B)44.031.583.53.633.0
Preservation (Pre-B)73.067.591.55.69.6

The results show substantial agreement between human judgments and our automatic evaluation pipeline. The relatively low false-positive rates indicate that the pipeline rarely identifies unsuccessful repairs as successful. However, the automatic evaluation is more conservative than human judgment, particularly for recovery, where the automatic success rate is 31.5% compared with 44.0% under human evaluation, with a false-negative rate of 33.0%. This discrepancy suggests that some semantically correct repairs are not fully captured by structural property matching. Nevertheless, the high overall agreement supports the reliability of our automatic metrics, while highlighting their tendency to underestimate successful recovery rather than overestimate agent performance.

Weakness 3: Limited Qualitative Visual Examples. The paper lacks sufficient qualitative examples illustrating different types of chart defects and their corresponding before-and-after repair results. Providing more visual comparisons across different error categories would help readers better understand the dataset composition and evaluation methodology.
Answer: We have added a new qualitative analysis in Appendix C.1 of the newly uploaded paper, presenting representative before-and-after visualizations across three defect categories: color contradiction, dropped series, and wrong chart type. Each example includes the ground-truth visualization, the flawed input, and the agent's final repair. These examples illustrate how different types of visualization defects manifest and how agents successfully correct them while preserving the intended content. They also demonstrate that semantically correct repairs need not exactly reproduce the reference visualization. We believe these additional examples provide a clearer understanding of the benchmark's defect categories and evaluation methodology.

Weakness 4: Lack of Fine-Grained Failure Case Analysis. Although the paper presents quantitative results across different defect categories and numbers of defects, it provides limited analysis of specific failure cases and the agent's diagnosis, repair, and verification processes. Including visualizations of representative failure trajectories would help clarify the underlying reasons for the agent's failures.
Answer: To provide a more fine-grained understanding of agent failures, we have added Appendix C.2 in the newly uploaded paper, which presents three representative failure trajectories illustrating distinct challenges in the diagnosis–repair–verification process. Specifically, the examples demonstrate: (1) an incomplete diagnosis that leads to partial repair, where the agent restores a missing series but overlooks axis truncation; (2) a regression during iterative repair, where the agent initially produces a correct visualization but subsequently removes a necessary legend; and (3) a verification failure, where the agent corrects a visible color inconsistency but fails to identify an incorrect numerical reconstruction. For each case, we present the original visualization, flawed input, and repair outcome, together with an analysis of the agent's intermediate actions and verification behavior. These examples reveal how failures can arise from incomplete diagnosis, unintended modifications, and insufficient verification, providing qualitative insights that complement our quantitative evaluation.

[D1] Lo, Leo Yu‐Ho, et al. "Misinformed by visualization: What do we learn from misinformative visualizations?", Computer Graphics Forum, 2022.

[D2] Tonglet, Jonathan, et al. "Is this chart lying to me? Automating the detection of misleading visualizations." Annual Meeting of the Association for Computational Linguistics, 2026.

Reviewer_ammYOpenReview ↗

Response to author

Hi, apologies for my late response.

Thank you for the additional experiments and clarifications. While they address some of my concerns, I still have reservations about the overall persuasiveness of the experimental evidence.

Reviewer_WpaJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VisAudit tests whether an agent can review a chart on its own, meaning find an unstated defect, fix it, leave the correct parts alone, and recognize when nothing needs fixing. It has 1,900 flawed instances from 854 ChartNet charts across 21 chart types and 10 flaw categories, plus 300 correct charts, split into three tracks that respectively give the defect, hide it, and mix in correct charts. Scoring is property based rather than judge based, using recovery over the perturbation footprint, preservation over untouched properties, and a correct chart recognition rate.

Critical strengths
  1. The no op case is the new idea and it earns its place. Every prior chart benchmark tells the model what to do, so none of them can catch a model that confidently edits something that was already fine, and recognition rates ranging from 33 to 72 percent show this separates systems that look similar on repair alone.

  2. The construction is careful in exactly the place synthetic defect injection usually breaks. Every generation prompt carries an evidence contract requiring the untouched table or summary to verifiably contradict the injected flaw, with explicit decline conditions and warnings about edits that leave the render unchanged. Scoring against extracted figure properties rather than a model judge also makes the numbers reproducible.

Critical weaknesses
  1. The model roster cannot support the claim. Every system evaluated is small or fast tier, with nothing above Qwen3-VL-235B and a lineup otherwise made of Haiku, Flash and 8B to 31B models, and the two ablations use only the weakest two. The conclusion that current multimodal agents are far from reliable review is therefore untestable from this data, since a capacity floor and a capability ceiling look identical.

  2. Preservation rewards doing nothing, which breaks the Track C analysis. Table 10 shows it plainly, with both models scoring 0.0 recovery and 100.0 preservation at a three turn budget. The same pattern shows up in Track C, where Qwen has the best recognition at 72.0 and nearly the worst flawed subset recovery at 26.4, which reads as conservatism rather than better verification, and with recovery and recognition in separate tables and no joint metric there is no way to tell the two apart.

Questions for authors
  1. Can you add two frontier models to all three tracks? If 47.4 percent holds the paper gets much stronger, and if it does not the conclusion needs rewriting.

  2. Please give one metric for Track C that penalizes both missed defects and needless edits. A do nothing baseline currently scores 100 on recognition and 100 on binary preservation.

  3. Rec-B needs every affected property fixed, so part of the multi flaw decline is just conjunction. Qwen at 53.0 / 23.0 / 17.8 roughly tracks a p cubed baseline while Gemini at 60.5 / 46.8 / 39.7 clearly beats it, so please report per flaw recovery as well.

  4. How were the 300 correct charts verified? They get only executability and agreement checks, not the four model panel used for the flawed set, and if any carry real defects then CCR penalizes models for being right.

3 comments

Thank you for your constructive comments. We believe that our discussion will lead to a better paper in general. We address your comments in a point-by-point manner.

Weakness 1: The model roster cannot support the claim. Every system evaluated is small or fast tier, with nothing above Qwen3-VL-235B and a lineup otherwise made of Haiku, Flash and 8B to 31B models, and the two ablations use only the weakest two. The conclusion that current multimodal agents are far from reliable review is therefore untestable from this data, since a capacity floor and a capability ceiling look identical.
Answer: To evaluate whether our findings generalize to stronger frontier multimodal models, we conducted additional experiments with Claude Opus 5.5 and GPT-6.1 Sol across all three benchmark tracks. For each track, we selected 300 instances through stratified sampling to preserve the distribution of single-flaw, double-flaw, and triple-flaw cases. We report joint success (JS), a stringent metric requiring the generated visualization to simultaneously satisfy executability, complete recovery (Rec-B), and complete preservation (Pre-B).

ModelTrack ATrack BTrack C
Claude Opus 5.575.057.359.5
GPT-6.1 Sol70.340.735.5

The results show that both frontier models substantially improve upon the previously evaluated models, particularly Claude Opus 5.5, which achieves joint success rates of 75.0%, 57.3%, and 59.5% across Tracks A, B, and C, respectively. GPT-6.1 Sol achieves 70.3%, 40.7%, and 35.5%. Nevertheless, even the strongest model fails to achieve joint success on 42.7% of Track B instances and 40.5% of Track C flawed instances. These findings demonstrate that stronger frontier models substantially advance autonomous visualization repair but still exhibit considerable limitations in reliably correcting all defects while preserving unaffected chart properties.

Weakness 2: Preservation rewards doing nothing, which breaks the Track C analysis. Table 10 shows it plainly, with both models scoring 0.0 recovery and 100.0 preservation at a three turn budget. The same pattern shows up in Track C, where Qwen has the best recognition at 72.0 and nearly the worst flawed subset recovery at 26.4, which reads as conservatism rather than better verification, and with recovery and recognition in separate tables and no joint metric there is no way to tell the two apart.
Answer: To address this concern, we introduce Open-World Success Rate (OWSR), a joint metric that evaluates an agent's ability to both repair flawed charts and avoid unnecessary modifications to initially correct charts. Specifically, for each flawed instance, success requires simultaneously satisfying executability, complete recovery (Rec-B), and complete preservation (Pre-B). For each initially correct instance, success requires correctly leaving the original visualization code unchanged (CCR). OWSR is defined as

OWSR=Nflawed⋅JS+Nclean⋅CCRNflawed+Nclean,\text{OWSR}=\frac{N_{\text{flawed}}\cdot\text{JS}+N_{\text{clean}}\cdot\text{CCR}}{N_{\text{flawed}}+N_{\text{clean}}},

where JS denotes joint success (of executability, Rec-B, and Pre-B) on flawed instances. We compute OWSR over all 2,200 Track C instances, consisting of 1,900 flawed and 300 initially correct charts. This metric penalizes both missed defects and unnecessary modifications. In particular, an agent that always leaves charts unchanged achieves only 13.6% OWSR, despite attaining 100% CCR and preservation.

ModelFlawed SuccessCCROWSR
Gemma21.735.023.5
Qwen20.872.027.8
Ministral18.633.020.6
Claude Haiku24.861.029.8
Mistral Large29.832.730.2
DeepSeek38.772.743.3
Gemini38.876.744.0

The results show that OWSR ranges from 20.6% to 44.0% across the evaluated models. Gemini achieves the highest OWSR (44.0%), followed by DeepSeek (43.3%). Notably, although Qwen achieves a relatively high CCR (72.0%), its OWSR is only 27.8%, lower than Mistral Large (30.2%) and Claude Haiku (29.8%). This demonstrates that high correct-chart recognition alone does not necessarily translate into strong open-world performance, as conservative behavior may also lead to missed repairs. Overall, OWSR provides a more comprehensive measure of open-world visualization review by jointly accounting for successful repair and appropriate non-intervention, rather than rewarding either recovery or preservation in isolation.

Question 1: Can you add two frontier models to all three tracks? If 47.4 percent holds the paper gets much stronger, and if it does not the conclusion needs rewriting.
Answer: Please refer to our answer to Weakness 1.

Question 2: Please give one metric for Track C that penalizes both missed defects and needless edits. A do nothing baseline currently scores 100 on recognition and 100 on binary preservation.
Answer: Please refer to our answer to Weakness 2.

Question 3: Rec-B needs every affected property fixed, so part of the multi flaw decline is just conjunction. Qwen at 53.0 / 23.0 / 17.8 roughly tracks a p cubed baseline while Gemini at 60.5 / 46.8 / 39.7 clearly beats it, so please report per flaw recovery as well.
Answer: We conducted an additional human evaluation of per-flaw recovery on 200 stratified Claude Haiku outputs from Track B, preserving the distribution of single-flaw, double-flaw, and triple-flaw instances. Human annotators independently assessed whether each injected defect was completely corrected in the final visualization, allowing semantically valid alternative repairs. Unlike Rec-B, which requires complete recovery of all affected properties, per-flaw recovery measures the proportion of individual injected defects successfully corrected.

GroupInstancesDefects EvaluatedDefects CorrectedRecovery (%)
Single-flaw42422969.1
Double-flaw9519013973.2
Triple-flaw6318914275.1
Overall20042131073.6

The per-flaw recovery rates are 69.05%, 73.16%, and 75.13% for single-flaw, double-flaw, and triple-flaw instances, respectively, with an overall recovery rate of 73.63% across 421 evaluated defects. These results show that individual flaw recovery does not decrease as the number of simultaneous flaws increases in this sample, despite the decline in complete recovery (Rec-B) observed in Table 8. This supports our original interpretation that the decrease in Rec-B largely reflects its all-or-nothing requirement rather than necessarily indicating that individual defects become more difficult to repair. The additional per-flaw analysis complements Rec-B and Rec-S by distinguishing complete chart recovery from the successful correction of individual defects.

Question 4: How were the 300 correct charts verified? They get only executability and agreement checks, not the four model panel used for the flawed set, and if any carry real defects then CCR penalizes models for being right.
Answer: We clarify that the 300 initially correct charts were validated using an automatic procedure rather than the four-model judging panel used for flawed instances. Specifically, we verified that the plotting code executed successfully and that the numerical values in the rendered charts matched the source data in table.csv. We additionally manually inspected 24 charts against their source tables and intended summaries, identifying two instances with incorrect colors, which were subsequently replaced. We acknowledge that the intended summaries were not independently checked for all 300 charts.
To assess whether potential inaccuracies in the initially correct charts affect our conclusions, we conducted an additional robustness analysis using a stricter numerical consistency criterion, requiring every numerical value in the rendered chart to appear in table.csv. Of the 300 initially correct charts, 236 satisfy this criterion. We recomputed CCR for all models on this validated subset and found that CCR increases by approximately 1–5 percentage points across models, while their relative ranking remains unchanged. These results suggest that our conclusions regarding correct-chart recognition are robust to this additional numerical consistency check.

Thank you for the responses. OWSR and the 13.6% no-op baseline directly resolve my concern about preservation rewarding inaction. However, two issues remain. The per-flaw recovery rests on a single backbone on Track B, which is too narrow to support the general claim. The CCR robustness check shows a systematic 1 to 5 point shift on the 236-chart subset. Noting that rankings are unchanged does not address the biased CCR and OWSR values. The intended summaries for the 300 correct charts were also not independently verified, and this is precisely the basis on which CCR labels a chart correct. I keep my rating.

Reviewer_pQz7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VisAudit, a benchmark for evaluating multimodal agents' ability to autonomously diagnose, repair, and verify data visualizations. Unlike existing benchmarks that primarily focus on chart generation or instruction-guided editing, VisAudit evaluates the complete diagnosis-repair-verification workflow in an interactive environment. The benchmark consists of three tracks with increasing levels of autonomy: diagnosed repair, autonomous repair, and open-world verification. It includes 1,900 flawed charts spanning 21 chart types and 10 flaw categories, along with 300 initially correct charts. The authors evaluate seven multimodal models using metrics covering recovery, preservation, executability, and correct-chart recognition.

Critical strengths
  1. The paper goes beyond conventional instruction-guided chart editing by evaluating whether agents can autonomously identify visualization defects, perform targeted repairs, and determine when no intervention is necessary. The three tracks provide a useful progression from diagnosed repair to autonomous repair and open-world verification.
  2. VisAudit covers 21 chart types and 10 flaw categories, including scenarios with multiple simultaneous defects. The controlled perturbation pipeline, automatic validation, cross-family LLM judging, and human agreement study provide a systematic foundation for constructing recoverable visualization errors.
Critical weaknesses
  1. Although VisAudit is presented as a benchmark for multimodal agents, the experiments do not isolate the contribution of visual information. The modality ablation in Table 9 removes code, table, and summary, but never removes the chart image itself. Since agents have access to both the visualization code and a detailed textual summary, it is unclear how much of the observed performance depends on visual understanding rather than text- and code-based reasoning.
  2. The benchmark emphasizes a diagnosis-repair-verification loop, but the evaluation primarily measures the correctness of the final artifact. CCR evaluates whether agents refrain from modifying initially correct charts, rather than whether they can recognize unsuccessful repairs. The paper does not establish whether agents actually inspect their rendered outputs, detect remaining errors after an intervention, or successfully correct those errors through subsequent iterations. Consequently, the experiments provide limited evidence about the effectiveness of the verification stage itself.
  3. Recovery and preservation are computed by comparing extracted figure properties against a reference visualization. However, a visualization can be semantically correct without reproducing all properties of the reference chart. The current evaluation does not establish how well these structural metrics align with human judgments of repair correctness, particularly when agents reconstruct charts without access to the original code.
Questions for authors
  1. Can the authors conduct a visual-modality ablation? Specifically, please evaluate agents without access to the initial chart image while retaining the code, table, and summary. A separate ablation disabling rendered-image feedback during iterative repair would help distinguish the contribution of initial visual understanding from visual verification. Please also report how frequently agents invoke the image-viewing tool.
  2. Can the authors directly evaluate diagnosis and verification accuracy? One possible experiment is to require agents to identify the suspected flawed components before editing and evaluate their predictions against the injected defects. For verification, the authors could analyze whether agents recognize unsuccessful intermediate repairs and whether subsequent iterations actually correct the remaining errors. Comparing performance with and without post-execution visual feedback would further quantify the value of self-verification.
  3. Can the authors validate the automatic evaluation metrics against human judgments? Please conduct a human evaluation on a stratified subset of repaired visualizations, particularly those produced without access to the original plotting code. Reporting the agreement between human judgments and Rec-B/Pre-B would help establish whether the metrics accurately capture successful repairs and distinguish incorrect outputs from valid alternative implementations.
3 comments

Thank you for your constructive comments. We believe that our discussion will lead to a better paper in general. We address your comments in a point-by-point manner.

Weakness 1: Although VisAudit is presented as a benchmark for multimodal agents, the experiments do not isolate the contribution of visual information. The modality ablation in Table 9 removes code, table, and summary, but never removes the chart image itself. Since agents have access to both the visualization code and a detailed textual summary, it is unclear how much of the observed performance depends on visual understanding rather than text- and code-based reasoning.
Answer: We are currently working on this additional ablation experiment and will report the performance once we obtain the results.

Weakness 2: The benchmark emphasizes a diagnosis-repair-verification loop, but the evaluation primarily measures the correctness of the final artifact. CCR evaluates whether agents refrain from modifying initially correct charts, rather than whether they can recognize unsuccessful repairs. The paper does not establish whether agents actually inspect their rendered outputs, detect remaining errors after an intervention, or successfully correct those errors through subsequent iterations. Consequently, the experiments provide limited evidence about the effectiveness of the verification stage itself.
Answer: We agree that evaluating the correctness of final visualizations alone does not fully characterize agents' diagnosis and verification capabilities. To address this concern, we conducted an additional human evaluation of 200 agent trajectories from Claude Haiku on Track B to directly examine the diagnosis–repair–verification process.
Specifically, human annotators examined agent trajectories to assess four aspects: (1) diagnosis accuracy, the proportion of trajectories in which agents correctly identify all original defects before editing; (2) visual inspection, the proportion of eligible repair attempts followed by inspection of the rendered visualization; (3) failure recognition, the proportion of unsuccessful repair attempts for which agents correctly identify all remaining defects; and (4) subsequent correction, the proportion of recognized unsuccessful repairs for which agents subsequently produce a fully correct visualization in a later iteration. For diagnosis accuracy, annotators compared agents' initial diagnoses against the ground-truth injected defects. For verification, annotators independently assessed intermediate and final repair correctness and examined agents' subsequent observations, assessments, and corrective actions.

Diagnosis Accuracy (%)Visual Inspection (%)Failure Recognition (%)Subsequent Correction (%)
45.5100.02.1775.0

The results reveal a substantial gap between visual inspection and effective verification. Although agents inspect rendered visualizations after every eligible repair attempt (100.0%), they correctly diagnose all original defects in only 45.5% of trajectories and recognize all remaining defects in just 2.17% of unsuccessful repair attempts. This indicates that visual inspection alone does not ensure effective verification, as agents frequently fail to identify errors that persist after their interventions. Notably, among the unsuccessful repairs that agents correctly recognize, 75.0% are subsequently corrected, suggesting that agents can often complete the repair once the remaining defects are accurately identified. Overall, these findings indicate that recognizing unsuccessful repairs is a major bottleneck in the diagnosis–repair–verification process.

Weakness 3: Recovery and preservation are computed by comparing extracted figure properties against a reference visualization. However, a visualization can be semantically correct without reproducing all properties of the reference chart. The current evaluation does not establish how well these structural metrics align with human judgments of repair correctness, particularly when agents reconstruct charts without access to the original code.
Answer: To assess whether our automatic evaluation metrics align with human judgments of repair quality, we conducted an additional human evaluation on 200 outputs generated by Claude Haiku. The samples were selected through stratified sampling to match the benchmark's distribution of single-flaw, double-flaw, and triple-flaw instances.
The table below compares the number of successful instances identified by human annotators and our automatic evaluation pipeline, together with their instance-level agreement rates and false-positive/false-negative rates.

MetricHumanAutomaticAgreement (%)FPR (%)FNR (%)
Recovery (Rec-B)44.031.583.53.633.0
Preservation (Pre-B)73.067.591.55.69.6

The results show substantial agreement between human judgments and our automatic evaluation pipeline. The relatively low false-positive rates indicate that the pipeline rarely identifies unsuccessful repairs as successful. However, the automatic evaluation is more conservative than human judgment, particularly for recovery, where the automatic success rate is 31.5% compared with 44.0% under human evaluation, with a false-negative rate of 33.0%. This discrepancy suggests that some semantically correct repairs are not fully captured by structural property matching. Nevertheless, the high overall agreement supports the reliability of our automatic metrics, while highlighting their tendency to underestimate successful recovery rather than overestimate agent performance.

Question 1: Can the authors conduct a visual-modality ablation? Specifically, please evaluate agents without access to the initial chart image while retaining the code, table, and summary. A separate ablation disabling rendered-image feedback during iterative repair would help distinguish the contribution of initial visual understanding from visual verification. Please also report how frequently agents invoke the image-viewing tool.
Answer: We are currently working on this additional ablation experiment and will report the performance once we obtain the results.

Question 2: Can the authors directly evaluate diagnosis and verification accuracy? One possible experiment is to require agents to identify the suspected flawed components before editing and evaluate their predictions against the injected defects. For verification, the authors could analyze whether agents recognize unsuccessful intermediate repairs and whether subsequent iterations actually correct the remaining errors. Comparing performance with and without post-execution visual feedback would further quantify the value of self-verification.
Answer: Please refer to our answer to Weakness 2.

Question 3: Can the authors validate the automatic evaluation metrics against human judgments? Please conduct a human evaluation on a stratified subset of repaired visualizations, particularly those produced without access to the original plotting code. Reporting the agreement between human judgments and Rec-B/Pre-B would help establish whether the metrics accurately capture successful repairs and distinguish incorrect outputs from valid alternative implementations.
Answer: Please refer to our answer to Weakness 3.

Thank you for the detailed response and the additional human evaluations. However, the visual-modality ablation, which is central to the paper’s claim of evaluating multimodal agents, is still pending, so I will keep my original score.

Reviewer_LDqxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VisAudit, a benchmark for evaluating whether multimodal agents can diagnose and repair flawed visualizations while preserving correct content and deciding when no intervention is needed. The benchmark contains 1,900 flawed instances created from 854 source charts, covering ten flaw categories, and 300 initially correct charts. It defines diagnosed-repair, autonomous-repair, and mixed-input verification tracks. The evaluation separately reports executability, recovery of perturbed properties, preservation of unaffected properties, and recognition of correct charts. Experiments with seven models and a shared interactive environment examine the effects of defect information, input modalities, and interaction budgets.

Critical strengths
  1. The benchmark addresses an important end-to-end task. Separating known-defect repair, autonomous diagnosis, and the decision not to modify a correct chart provides a useful framework for studying visualization review beyond chart generation or instruction-following editing.

  2. The evaluation distinguishes repair from preservation. Reporting recovery and preservation separately makes it possible to identify agents that fix an intended defect while changing previously correct chart properties. The controlled perturbations and shared instances across tracks also support targeted analyses of diagnosis and repair.

Critical weaknesses
  1. The validity of the benchmark as a measure of real-world visualization review remains uncertain. Most flawed instances are generated through controlled code perturbations, and evaluation relies on agreement with extracted reference figure properties. This makes the benchmark reproducible, but it is unclear how well the injected errors represent naturally occurring problems or whether semantically correct alternative repairs are scored appropriately. The reported human check of generated tasks is useful, but it does not by itself validate the scoring of actual agent repairs.

  2. The reported agent limitations may depend substantially on the evaluation harness and metrics. All models are tested in a shared interactive environment, so the results reflect the combination of model, prompts, tool interface, and execution budget. The paper does not establish how robust its conclusions are across stronger or alternative general-purpose agent frameworks. In addition, the headline recovery rate does not establish how often an agent simultaneously executes successfully, recovers all affected properties, and preserves the unaffected ones.

Questions for authors
  1. Evaluator validity: Have the authors independently audited actual agent outputs with human judges, including alternative but semantically correct repairs? Please report agreement and false-positive/false-negative rates for recovery and preservation, not only agreement on whether generated tasks contain a valid injected defect.

  2. Joint success: What fraction of instances simultaneously satisfy executability, complete recovery, and complete preservation? How are technical failures and uncompleted runs included in the denominator?

  3. Agent framework sensitivity: How do results change with a different general-purpose agent harness or a stronger task-adapted baseline under comparable model and tool budgets? Can the authors separate limitations of the underlying model from limitations induced by the shared interaction protocol?

  4. Real-world and clean-chart validity: Can the authors evaluate naturally occurring visualization defects or matched clean and perturbed versions of the same source charts? Since clean and flawed charts use disjoint sources, how do the authors rule out source or chart-distribution cues that are unrelated to recognizing a defect?

3 comments

Thank you for your constructive comments. We believe that our discussion will lead to a better paper in general. We address your comments in a point-by-point manner.

Weakness 1: The validity of the benchmark as a measure of real-world visualization review remains uncertain. Most flawed instances are generated through controlled code perturbations, and evaluation relies on agreement with extracted reference figure properties. This makes the benchmark reproducible, but it is unclear how well the injected errors represent naturally occurring problems or whether semantically correct alternative repairs are scored appropriately. The reported human check of generated tasks is useful, but it does not by itself validate the scoring of actual agent repairs.
Answer: The ten flaw categories considered in VisAudit are grounded in visualization errors documented in prior studies of real-world charts. In particular, [A1] analyzed over 1,000 real-world misleading visualizations and identified 74 types of visualization issues, including incorrect or misleading labels, missing legends, plotting errors, inappropriate chart types, problematic axis ranges, and misleading visual encodings, which closely correspond to the flaw categories considered in VisAudit. Controlled synthetic generation is also adopted in existing visualization benchmarks, such as Misviz [A2], which includes synthetically generated misleading charts alongside real-world examples. Our controlled perturbations are therefore intended to instantiate documented real-world classes of visualization errors while enabling systematic evaluation with precisely identified defects and reference properties.
Human evaluation. To assess whether our automatic evaluation metrics align with human judgments of repair quality, we conducted an additional human evaluation on 200 outputs generated by Claude Haiku. The samples were selected through stratified sampling to match the benchmark's distribution of single-flaw, double-flaw, and triple-flaw instances. Unlike our original human validation of generated benchmark tasks, this evaluation directly examines actual agent-generated repairs. Human annotators independently assessed whether each repaired visualization successfully corrected all original defects (recovery) and preserved all unaffected chart components (preservation). Importantly, human judgments were based on semantic correctness, allowing valid alternative repairs that do not exactly reproduce the reference visualization.
The table below compares the number of successful instances identified by human annotators and our automatic evaluation pipeline, together with their instance-level agreement rates and false-positive/false-negative rates.

MetricHumanAutomaticAgreement (%)FPR (%)FNR (%)
Recovery (Rec-B)44.031.583.53.633.0
Preservation (Pre-B)73.067.591.55.69.6

The results show substantial agreement between human judgments and our automatic evaluation pipeline. The relatively low false-positive rates indicate that the pipeline rarely identifies unsuccessful repairs as successful. However, the automatic evaluation is more conservative than human judgment, particularly for recovery, where the automatic success rate is 31.5% compared with 44.0% under human evaluation, with a false-negative rate of 33.0%. This discrepancy suggests that some semantically correct repairs are not fully captured by structural property matching. Nevertheless, the high overall agreement supports the reliability of our automatic metrics, while highlighting their tendency to underestimate successful recovery rather than overestimate agent performance.

Weakness 2: The reported agent limitations may depend substantially on the evaluation harness and metrics. All models are tested in a shared interactive environment, so the results reflect the combination of model, prompts, tool interface, and execution budget. The paper does not establish how robust its conclusions are across stronger or alternative general-purpose agent frameworks. In addition, the headline recovery rate does not establish how often an agent simultaneously executes successfully, recovers all affected properties, and preserves the unaffected ones.
Answer: We have conducted additional analyses to assess the sensitivity of our results to the agent harness and to evaluate whether agents simultaneously satisfy all requirements for successful visualization repair.
Agent harness sensitivity. To investigate whether the observed limitations arise from our shared interaction protocol, we additionally evaluate Qwen3-VL-235B-A22B-Instruct on Track B using deepagents [A3], an open-source general-purpose agent harness built on LangChain. Unlike our original lightweight tool-calling agent, deepagents provides additional agent orchestration capabilities, including planning and context management. To ensure a controlled comparison, we retain the same underlying Qwen model, benchmark instances, input modalities, visualization tools, execution environment, and evaluation pipeline, while replacing the agent orchestration with deepagents. The results are summarized below.

Agent HarnessExec.Rec-BPre-B
VisAudit (Original)99.527.766.7
deepagents99.024.570.1

The results show that the two agent harnesses achieve comparable performance. Importantly, by keeping the underlying Qwen model, benchmark instances, input modalities, tools, and execution environment fixed while varying the agent harness, this comparison allows us to assess the contribution of agent orchestration to the observed limitations. The relatively small performance differences indicate that replacing our original harness with an established general-purpose alternative does not substantially improve recovery, suggesting that the observed difficulties are not primarily caused by implementation-specific limitations of our original interaction protocol. Instead, the limited recovery performance persists across both harnesses, pointing to challenges associated with the underlying model's capabilities under the evaluated settings.
Joint success evaluation. We introduce a joint success metric, which requires an output to simultaneously satisfy executability, complete recovery (Rec-B), and complete preservation (Pre-B). We report the joint success of all models on the flawed dataset (with 1,900 flawed instances).

ModelTrack ATrack BTrack C
Claude Haiku60.925.524.8
Qwen57.622.120.8
Mistral Large52.527.529.8
Gemma52.428.421.7
Ministral28.515.918.6
DeepSeek61.338.737.1
Gemini67.739.338.6

The joint success results further demonstrate the challenges of autonomous visualization repair. Although the evaluated models achieve relatively high executability, their joint success rates are substantially lower, indicating that producing executable code does not necessarily result in complete and faithful repairs. Gemini achieves the highest joint success across all three tracks, with 67.7% on Track A, 39.3% on Track B, and 38.6% on Track C. The substantial decline from Track A to Track B highlights the difficulty of autonomous defect diagnosis, while the generally comparable results between Tracks B and C suggest that introducing uncertainty about whether intervention is necessary does not uniformly reduce repair success. Overall, these findings reinforce that reliable visualization repair requires simultaneously correcting all defects and preserving unaffected properties, rather than satisfying either criterion in isolation.

Question 1: Evaluator validity: Have the authors independently audited actual agent outputs with human judges, including alternative but semantically correct repairs? Please report agreement and false-positive/false-negative rates for recovery and preservation, not only agreement on whether generated tasks contain a valid injected defect.
Answer: Please refer to our response to Weakness 1, where we provide additional human evaluation of 200 agent-generated repairs, including agreement rates and false-positive/false-negative rates for both recovery and preservation.

Question 2: Joint success: What fraction of instances simultaneously satisfy executability, complete recovery, and complete preservation? How are technical failures and uncompleted runs included in the denominator?
Answer: Please refer to our response to Weakness 2 for joint success. Regarding the denominator, we include all assigned benchmark instances, including technical failures and incomplete runs. Such instances are assigned a success value of 0 rather than being excluded from the evaluation.

Question 3: Agent framework sensitivity: How do results change with a different general-purpose agent harness or a stronger task-adapted baseline under comparable model and tool budgets? Can the authors separate limitations of the underlying model from limitations induced by the shared interaction protocol?
Answer: Please refer to our response to Weakness 2 for the detailed comparison with deepagents, an alternative general-purpose agent harness.

Question 4: Real-world and clean-chart validity: Can the authors evaluate naturally occurring visualization defects or matched clean and perturbed versions of the same source charts? Since clean and flawed charts use disjoint sources, how do the authors rule out source or chart-distribution cues that are unrelated to recognizing a defect?
Answer: Thank you for this suggestion. To investigate potential source-distribution bias between clean and flawed charts in Track C, we conducted an additional matched-source experiment using Claude Haiku. Specifically, we replaced the original 300 initially correct charts with 300 clean visualizations drawn from the same source charts used to construct the flawed evaluation set. We retained the original 1,900 flawed instances and the same Track C evaluation protocol, ensuring that the only change was the source distribution of the initially correct charts.
The following table compares the correct-chart recognition rate (CCR) under the original and matched-source settings.

SettingCCR (%)
Original clean charts61.0
Matched-source clean charts42.0

The results show that CCR decreases from 61.0% under the original setting to 42.0% under the matched-source setting, a reduction of 19.0 percentage points. This indicates that the source distribution of initially correct charts can substantially influence recognition performance, suggesting that the original CCR results may partly reflect source-distribution differences rather than defect recognition alone. Nevertheless, the matched-source evaluation removes source identity as a systematic distinction between the clean and flawed sets, providing a more controlled assessment of correct-chart recognition. We will clarify this sensitivity in the revised paper and include the matched-source evaluation as an additional robustness analysis.

[A1] Lo, Leo Yu‐Ho, et al. "Misinformed by visualization: What do we learn from misinformative visualizations?", Computer Graphics Forum, 2022.

[A2] Tonglet, Jonathan, et al. "Is this chart lying to me? Automating the detection of misleading visualizations." Annual Meeting of the Association for Computational Linguistics, 2026.

[A3] https://github.com/langchain-ai/deepagents

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.