VisualTraceBench: A Human-Performance Benchmark for Multimodal AI SQL Plan Debugging with PlanViz Visual Traces

We introduce VisualTraceBench, a curated dataset of 192 production SQL performance regression bug reports from a large enterprise column-store database system (2016–2025), each annotated with the diagnostic artifact attached to the bug report: a PlanViz-style SQL execution plan visualization (an operator tree with per-operator cost annotations) versus tabular SQL profiler output (rows of operator + numeric cost without spatial layout). From this dataset we derive VPER (Visualization Performance Enhancement Rate), a metric quantifying the efficiency gain that visual SQL plan artifacts provide over tabular profiler tables during human debugging: our empirical measurement yields VPER = 27.9% (median normalized efficiency ratio 0.464 for visual+tabular vs. 0.643 for tabular-only, acceleration multiplier 2.16x vs. 1.56x; Mann-Whitney U test p = 0.689; small-sample limitation acknowledged). We formalize the AI-VPER benchmark protocol, which evaluates multimodal AI systems (GPT-4V, Claude 3+, Gemini) on the same bug reports under matched text-vs-image conditions, using human VPER = 27.9% as the performance floor. The AI task is to identify the regressed SQL operator (HashJoin, TableScan, Aggregation, IndexSearch, etc.) given either a tabular profiler dump or a PlanViz operator-tree screenshot. VisualTraceBench is, to our knowledge, the first public dataset of real SQL performance regression bugs paired with visual execution plan artifacts; prior debugging benchmarks (SWE-bench, Defects4J) cover functional bugs without visual diagnostic artifacts, and no benchmark currently evaluates SQL plan visual reasoning. We release the benchmark specification, evaluation protocol, anonymized dataset, and baseline analyses to enable reproducible comparison of multimodal AI SQL plan debugging capabilities.