- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Visual Data Agent (VDA), an agentic framework that enables LLM-based data agents to actively visualize and inspect structured data rather than relying solely on textual records and numerical summaries. VDA integrates schema-based booting views, calibrated chart rendering, an observation window, agent-generated notes, visual memory, and coverage feedback to support iterative data exploration.
The authors also introduce DataPatternBench, a benchmark for detecting data defects and discovering distributional patterns in real multi-table databases. Experiments across multiple LLMs, agent frameworks, and public benchmarks demonstrate improvements in data analysis performance, particularly on DataPatternBench, where VDA improves detection F1 from 59.8% to 84.1% and discovery F1 from 43.1% to 78.0% over OpenHands with GPT-5.4.
Critical strengths
1. Practical problem and effective system design. The paper addresses an important limitation of existing data agents, which often rely on sampled records and numerical summaries that may obscure distributional patterns. VDA provides a coherent framework for incorporating visual observations into iterative data exploration, with substantial improvements on data defect detection and pattern discovery tasks.
2. Comprehensive experimental evaluation. The paper evaluates multiple LLMs, agent frameworks, and benchmarks, with extensive ablations and controlled comparisons of different information representations. The introduction of DataPatternBench also provides a useful evaluation setting with verifiable ground truth for data pattern discovery.
Critical weaknesses
1. Limited methodological novelty over existing visualization-based agents. The central idea of actively generating, inspecting, and iteratively refining visualizations has already been explored in prior work, such as AgentPoirot (InsightBench) and VESTA. The proposed components, including rule-based visualization initialization, agent notes, memory, and coverage feedback, largely build upon established techniques. Although their integration is practically useful, the paper does not sufficiently distinguish its methodological contributions from these closely related approaches. Moreover, the experiments primarily compare against general-purpose agent frameworks rather than specialized visualization-based agents, making it difficult to assess the significance of the proposed framework relative to existing work.
2. Insufficient isolation of the benefits of visual perception. The reported improvements may arise from better information coverage, structured exploration, and additional tool interactions rather than visual perception itself. In particular, the comparison with complete numerical text shows comparable performance to chart-based observations, with discovery F1 slightly favoring text (76.6% vs. 75.8%). This weakens the central claim that visual perception provides an advantage beyond information completeness. Furthermore, the additional computational and token costs are not sufficiently controlled when assessing the practical benefits of the framework.
Questions for authors
Q1. Novelty Clarification. Prior works such as AgentPoirot (InsightBench) have already investigated iterative workflows in which agents generate visualizations, interpret the results, and use them to guide subsequent analysis. What are the key methodological differences between VDA and these approaches? Which components constitute genuinely new contributions beyond integrating existing visualization and agent mechanisms?
Q2. Comparison with specialized baselines. Why are specialized visualization-based agents, such as AgentPoirot, not included in the experimental comparisons? The current baselines primarily consist of general-purpose agent frameworks. Can the authors provide comparisons with these more directly related approaches, or clarify whether their mechanisms can be adapted to the evaluated tasks?
Q3. Contribution of visual perception versus information completeness. The comparison with complete numerical text suggests that visual observations do not consistently outperform equally informative textual representations. How much of VDA's improvement can be attributed specifically to visual perception, rather than improved data coverage, exploration strategies, or additional tool interactions? Could the authors provide further controlled comparisons under comparable information and computational budgets?
Q4. Effectiveness of rule-based exploration. The ablation results suggest that booting views contribute substantially to performance, while applying predefined statistical checks to the same views already achieves 77.6% discovery F1, close to VDA's 78.0%. How much does the agent's active exploration contribute beyond systematic visualization and conventional statistical analysis? Could the authors compare VDA with a stronger automated statistical analysis pipeline using the same initial views?
Q5. Computational efficiency and practical scalability. The paper reports higher cumulative token consumption for VDA than for OpenHands, and chart-based observations are more expensive than numerical summaries in the fixed-view experiments. How do the performance gains compare under strictly matched token, tool-call, or monetary budgets? Does VDA remain advantageous under resource constraints representative of practical large-scale database analysis?
8 comments
Response to Reviewer s272 (Part 2)
W2 & Q3. Visual perception versus information completeness. We would like to clarify that the 76.6% vs. 75.8% comparison covers only the cases where the complete-text runs finished. On the full evaluation set, complete text reaches 64.6% discovery F1 and 77.6% detection F1, against 78.0% and 84.1% with charts (Table 11, Appendix D.1). Complete text fails 11 runs and uses about 40% more tokens, because its size grows with the data while a chart's cost stays fixed.
The paper also isolates visual perception under fixed views, where coverage, exploration and tool interactions are identical. Detection F1 is 70.7% with numerical summaries and 75.6% with charts (Section 3.3).
To better respond to the reviewer, we further ran this control on pattern discovery, the task that depends most on distributional patterns over time. We remove only the chart images from VDA and keep all other components and budgets unchanged.
| System (GPT-5.4) | Discovery F1 |
|---|---|
| VDA without chart images | 50.7 |
| VDA | 77.4 |
Removing the chart images lowers discovery F1 by 27 points (p = 0.008). Visual perception therefore contributes beyond data coverage, exploration strategies and tool interactions.
W2 & Q5. Matched budgets. We let OpenHands use as many tokens as used by VDA, on all 117 DataPatternBench cases with GPT-5.4.
| System (GPT-5.4) | Tokens per case | Detection F1 | Discovery F1 |
|---|---|---|---|
| OpenHands | 0.2M | 61.9 | 37.3 |
| OpenHands with more tokens | 1.3M | 64.8 | 46.5 |
| VDA | 1.3M | 76.7 | 75.1 |
With a similar number of requests and tokens, VDA still leads by 12 points on detection (p = 0.05) and 29 points on discovery (p = 0.004). For larger databases, a chart's cost stays fixed while text grows with the rows.
Q4. Contribution of active exploration beyond systematic views and statistics. We would like to clarify that the 77.6% result is not obtained under the same setting as VDA. It is a reference that is given the definitions of the targets in advance. Its purpose is to show the score that can be reached when the target definitions are known. It is obtained by applying the exact tests and thresholds that the benchmark uses to accept each target. VDA is not given these tests and still reaches the same score.
To answer the reviewer's questions, we ran two comparisons that start from the same initial views.
| Same initial views | Discovery F1 |
|---|---|
| OpenHands analyzing these views with SQL and Python, without charts | 58.6 |
| VDA without the plotting tool (Table 3) | 64.1 |
| VDA | 77.4 |
The agent's active exploration contributes 13 points beyond systematic visualization, and VDA scores 19 points above statistical analysis of the same views (p = 0.016). Active exploration is therefore a major source of VDA's gain, not a minor addition to systematic views and statistics.
We will add these comparisons and controls to the paper, and revise the related work as described above.
Dear Reviewer s272,
Thank you again for your detailed review. As the discussion period ends soon, we would like to kindly check whether you have any remaining questions, as we hope our additional experiments have addressed all of your concerns.
In summary, VDA differs fundamentally from existing plotting agents, and it outperforms AgentPoirot by 18 points on detection and 39 points on discovery at a similar token cost. The gain mainly comes from our proposed visual perception component, since removing only the charts lowers the performance significantly. It is not explained by extra compute either, because OpenHands with a similar number of tokens scores 12 points lower on detection and 29 points lower on discovery. Finally, active exploration cannot be replaced by rule-based views and statistical checks. VDA scores 13 points higher on discovery than without active exploration, and 19 points higher than an agent performing statistical analysis only.
We would be glad to clarify anything further. Thank you!
Response to Reviewer s272 (Part 1)
Thank you for the detailed questions, and for recognizing the practical problem, the system design and the comprehensive evaluation. We address each concern below with new experiments.
W1 & Q1. Novelty relative to AgentPoirot and VESTA. We would like to clarify that we do not claim data plotting itself as our main contribution. Instead, VDA contributes a way for the agent to perceive the database context through calibrated charts and to explore it systematically, which fundamentally differs from AgentPoirot and VESTA.
AgentPoirot generates analysis questions about a CSV dataset and writes code that draws a plot for each question. The base model never observes the plot. It reads the plotted values as text, cut to the first 50 values per axis. Any pattern beyond those values is dropped. By contrast, VDA improves this by aggregating all information into charts, and directly places charts in the model's context window. Each chart is computed from all relevant data and calibrated for better visibility, so the base model can attend to the corresponding data patterns at once.
VESTA and VDA solve different problems, based on different abilities and workflows. VESTA proposes a statistical model of the data, and each plot checks whether that model fits. Its key ability is to test and revise a hypothesis. By contrast, VDA focuses on long-horizon and context-heavy data tasks. It receives an unfamiliar database with an overwhelming number of tables and columns without a hypothesis. Its main challenge is planning how to consume this overwhelming data context. Specifically, it decomposes the exploration into many small steps and decides each step based on the plotted charts. In VESTA, a hypothesis decides what to plot. In VDA, the charts decide what to examine next, from booting views of the whole database to follow-up charts and checks with SQL or Python. Because each chart summarizes all rows of a column in a fixed number of tokens, the agent can take many such steps. Agent notes, visual memory and coverage feedback support these steps, so that the agent keeps earlier evidence and covers the whole database.
W1 & Q2. Comparison with a specialized baseline. We thank the reviewer for this suggestion. AgentPoirot can be adapted to our tasks, so we ran its official code on all 117 DataPatternBench cases with GPT-5.4. Since it analyzes one table at a time, we run it on every table and convert its insights into our findings format.
At a similar token cost, VDA leads AgentPoirot by 18 points on detection and 39 points on discovery (both p = 0.004). AgentPoirot draws many plots, but its model never sees them (Q1), so it performs close to OpenHands. It also does not outperform VDA on InsightBench, its own benchmark (50.2 vs 51.9).