acceptodds
Under review as a conference paper at ICLR 2027

What You Think Is What You Plot? A Benchmark for Decoding Visualization Intent from EEG

Abstract

Non-invasive Brain-Computer Interfaces (BCIs) have shown promise in decoding simple commands. Their capacity to interpret complex cognitive intents remains largely unexplored. A critical application is direct visualization generation, where a system can render charts simply by reading an analyst's mind. We introduce EEG2VIS, a novel task for decoding executable visualization (VIS) programs directly from non-invasive electroencephalography (EEG) signals. This task enables a “what you think is what you plot” paradigm for data exploration. Decoding visualization intent from EEG is intrinsically difficult due to the compositional nature of analytical queries and the low signal-to-noise ratio of scalp EEG. To facilitate research in this direction, we construct EEG2VIS-Bench, the first benchmark dataset comprising 9,531 rigorously aligned trials from 23 domain experts (46 human hours) under a three-stage cognitive paradigm. We evaluate a spectrum of baselines, ranging from end-to-end foundation models to LLM-aligned teacher-student architectures and a neuro-symbolic noisy-channel decoder. Extensive experiments show that EEG signals provide statistically significant information for coarse-grained intent, achieving up to 91.09% accuracy on chart type recognition. Accurately decoding deep compositional logic, such as data filtering conditions and sorting rules, remains a fundamental bottleneck, with independent clause-set accuracy dropping to 75.30%. We also demonstrate that current high full-query exact-match scores (62.59%) heavily rely on retrieval cache mechanisms (routing 58.05% of queries) rather than pure neural decoding.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, brain-computer interface, visualization, benchmark, compositional decoding, neural decoding, structured prediction, EEG2VIS

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V6MPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EEG2VIS-Bench, a dataset of 3,177 trials from 23 participants for decoding structured visualization queries from EEG. The authors propose a teacher–student framework with masked EEG pretraining, semantic alignment, and grammar-constrained decoding. Experiments achieve 27.68% full-program accuracy and demonstrate EEG signal dependence, but reveal limited generalization to unseen participants and programs.

Critical strengths
  1. Novel benchmark with objectively verifiable structured outputs. The paper introduces a new EEG-to-visualization decoding task and dataset, extending EEG decoding to executable structured programs. The use of component-level accuracy and full-program exact match enables more objective and fine-grained evaluation.

  2. EEG signal-dependence validation. The authors include multiple control experiments, particularly matched EEG shuffling, to assess whether the model relies on genuine EEG signals. The substantial performance gap between real EEG and shuffled EEG provides evidence that trial-specific EEG information contributes to prediction performance.

Critical weaknesses
  1. The paper does not clearly explain how schema-specific field names (e.g., region, sales) and filter values are generated, or whether the Data Table or Schema is available during inference. Key implementation details of the Teacher Representation, EEG Student architecture, alignment loss, and loss weighting are also insufficiently specified. Furthermore, the contributions of Grammar-constrained Decoding and the Schema Prior are not independently evaluated. Consequently, it is difficult to determine how much of the reported performance can be attributed to EEG decoding rather than external structural priors.

  2. The paper does not verify whether the visualization mentally represented by participants during query interpretation matches the predefined ground truth. Additionally, the high chart-type accuracy achieved using only Duration and Padding raises concerns about potential dataset biases. The substantial performance gap between seen and unseen programs suggests limited compositional generalization. Finally, the large discrepancy between Render Rating ≥ 4 (73.3%) and Full Program Exact Match (27.33%) lacks sufficient explanation or execution-level validation. These issues limit the strength of the paper's claims about decoding structured visualization intent from EEG.

Questions for authors
  1. How are schema-specific field names (e.g., region, sales) and filter values generated? Is the Data Table or Schema available during inference, or are these elements predicted directly from EEG? Could the authors also clarify the output spaces of the prediction heads, including the number of chart types and how clauses are represented?

  2. Figure 5 suggests that the Qwen3 Teacher receives both the Data Table and Text Query. How is the teacher representation constructed (e.g., selected layers, pooling)? Could the authors clarify the EEG Transformer architecture, the definition of the alignment loss, and the selection of loss weights (α\alpha and β\beta)? An ablation without teacher alignment would help establish its contribution.

  3. How are the valid program candidates and schema-conditioned prior in Equation (10) constructed? Could the authors report results for unconstrained decoding, grammar-constrained decoding, the full model, and Schema Prior Only? These comparisons are important for determining how much of the final performance is attributable to EEG rather than structural priors.

  4. Did the experimental protocol verify whether participants' mentally represented visualizations matched the predefined ground truth? Without comprehension checks or intent verification, how can the authors distinguish decoding actual visualization intent from predicting stimulus-associated labels?

  5. In Table 2, Duration alone achieves 51.54% chart-type accuracy, increasing to 58.73% with Padding. What classifiers are used for these baselines, and how can such high accuracy be explained? Furthermore, does the random split allow identical queries or visualization stimuli across training and test sets? Given the substantial performance drop on unseen programs (Table 4), could the authors clarify stimulus overlap and evaluate generalization under stricter splits?

  6. Why does Render Rating ≥ 4 reach 73.3% while Full Program Exact Match is only 27.33%? Do these cases represent semantically equivalent programs, or merely visually similar charts? Execution-based equivalence results or representative examples would help clarify this discrepancy.

  7. The title and abstract in the submitted PDF appear to differ from those on OpenReview.

Reviewer_AyKWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether brain activity recorded through EEG can be used to predict the data visualizations a person intends to create while interpreting analytical questions. The authors introduce EEG2VIS-Bench, a dataset that pairs EEG recordings from data-analysis experts with structured visualization programs specifying chart types, data fields, aggregations, and filtering conditions. They also develop a baseline model that combines EEG representation learning with knowledge distilled from a pretrained language model to decode these programs. Experiments show that the model substantially outperforms controls that remove or shuffle the EEG signal, providing evidence that EEG contains meaningful information about the intended visualization. However, performance deteriorates considerably when generalizing to new participants or previously unseen program structures. The paper's main contribution is establishing a benchmark for structured semantic decoding from EEG and demonstrating its initial feasibility.

Critical strengths

Novel benchmark for structured semantic decoding from EEG. The paper introduces an interesting new task that extends EEG-based language decoding to the prediction of executable visualization programs. Unlike conventional EEG-to-text tasks, the output requires jointly predicting chart types, data fields, aggregations, and logical constraints. The dataset is also carefully constructed, with separate recording stages for table reading, query interpretation, and visualization evaluation. This provides a useful testbed for studying whether compositional semantic information can be recovered from non-invasive brain signals.

Convincing evidence that EEG contains task-relevant predictive information. The experiments include several meaningful controls, particularly duration-matched EEG shuffling, which help address known concerns in prior EEG-to-text research about models relying primarily on language priors or superficial artifacts. The substantial improvement over these controls provides encouraging evidence that trial-specific EEG contributes useful information to structured prediction. The authors also evaluate generalization to unseen participants and program structures, revealing important limitations that make the benchmark informative for future research.

Critical weaknesses

Limited empirical characterization of structured semantic decoding. While the paper demonstrates that EEG contains useful predictive information about visualization programs, the experiments leave open how much structured semantic information is actually being recovered. In particular, full-program accuracy drops substantially on previously unseen programs, suggesting a strong dependence on training-program coverage. It would be useful to provide further empirical evidence of what the model learns, either through more targeted analyses of full-program generalization or through component-level evaluations that examine whether individual semantic elements can be recovered and recombined across different program structures. For example, the authors report 7.08% accuracy on unseen programs, but do not report the corresponding performance of simple baselines or EEG-shuffling controls, making it difficult to assess how meaningful this result is. Of course, achieving high accuracy on unseen programs is a challenging problem and need not be expected of an initial benchmark. However, more targeted empirical analysis would help clarify the extent to which the results demonstrate structured semantic decoding beyond recognizing familiar program patterns.

Questions for authors

Interpretation of the information recovered from EEG. The authors deserve credit for including EEG-shuffling and noise controls, which provide strong evidence that trial-specific EEG contributes predictive information. However, it remains unclear how much of this information reflects the semantics of the analytical query, as opposed to lower-level linguistic or behavioral signals associated with reading the displayed text. This distinction is relevant given prior concerns about what EEG-to-text models actually learn (e.g., Jo et al., 2024). While fully disentangling these factors may be beyond the scope of this paper, additional analyses using the existing dataset could help clarify what information is driving the predictions and better contextualize the results.

Reviewer_rB5eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EEG2VIS-Bench, containing 3,177 table-query-visualization trials from 23 participants with data-analysis experience. It studies prediction of structured visualization programs from EEG recorded while participants read and interpret displayed analytical queries. An EEG Transformer is first pretrained by reconstructing masked EEG from training-set table and chart epochs, then trained using program-component supervision and alignment to a text teacher. A grammar-constrained decoder combines neural predictions with a schema-conditioned prior. The model achieves 27.68% full-program exact match on a participant-overlapping test split, compared with 2.67% after length-matched EEG substitution.

Critical strengths
  • A structured (and useful) prediction task with a self-collected dataset. The authors introduce a dataset and evaluation setup for predicting executable visualization programs from EEG, requiring the model to identify chart type, data fields, aggregations, and filters. This provides a concrete way to assess whether predictions correctly answer the analytical question and to examine which components the model can recover reliably. The staged recording protocol, 23-participant collection, and pairing with canonical programs is also a contribution even beyond their reported decoding performance.

  • Meaningful signal analysis and transparency of limitations. For example, the 25.01-percentage-point full-match gap suggests that the trained predictor uses trial-specific context beyond duration and participant identity alone. Reporting duration/padding baselines, five-seed variation, participant transfer, and seen/unseen-program results substantially improves the evidential value of the paper. It's also appreciated that the authors acknowledge the generalization limitations (unseen vs seen, and shared vs not shared participants).

Critical weaknesses
  • The results provide limited evidence of generalizable decoding. In Table 4, 275/487 test examples have programs seen in training. Of 135 exact successes, 120 occur in that subset which are approximately 89% of successes. This is useful transparency, but the overall result therefore provides limited evidence for compositional generalization. Also, the authors show that the recording matters by replacing the EEG, but not why it matters. Is it actually because users produce different signals when interpreting a request for a bar versus a line chart, or could other features of the displayed text or recording explain the difference? Furthermore, the paper compares against nuisance-feature controls but not competitive EEG encoders under the same decoder, and it does not isolate the benefit of teacher alignment or the grammar prior. These omissions limit our understanding of the benefits of the proposed approach.

  • It is unclear how much of the prediction comes from EEG and how much is supplied by the decoder. Section 3 describes prediction from query-stage EEG, but Equation 10 also uses information about the table structure and a set of valid programs. The authors do not clearly explain how these programs are constructed, how many the model chooses between, or how specific field names and filter values are predicted. These details matter because choosing between a few candidate programs is a different task from constructing a complete program. It is also unclear whether the controls receive the same table information and candidate programs. A comparison using this information without EEG, together with a clearer description of the decoder, would help establish what the reported accuracy actually demonstrates.

Questions for authors
  • What information does the model receive when making a prediction? Besides EEG, does it receive the table structure or a list of candidate programs? How are these candidates constructed, and how many does it choose between? How well does the system perform with the same information but without EEG?
  • What is shared between the training and test sets? Do the same questions, tables, or stimulus triples appear in both? How was presentation order randomized? Can the authors report results for a split where all test programs are absent from training, together with the corresponding controls?
  • What do the different parts of the method contribute? How does the model perform without alignment to the text teacher? How does it compare with other EEG encoders using the same decoder? Also, does the teacher receive only the question or both the table and question?
  • Why is there such a large difference between program accuracy and visual correctness? On the same 150 predictions, 73.3% of charts are rated identical or only cosmetically different, but only 27.33% of programs match exactly. Can the authors show examples and explain which program differences account for this gap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.