- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EEG2VIS-Bench, a dataset of 3,177 trials from 23 participants for decoding structured visualization queries from EEG. The authors propose a teacher–student framework with masked EEG pretraining, semantic alignment, and grammar-constrained decoding. Experiments achieve 27.68% full-program accuracy and demonstrate EEG signal dependence, but reveal limited generalization to unseen participants and programs.
Critical strengths
-
Novel benchmark with objectively verifiable structured outputs. The paper introduces a new EEG-to-visualization decoding task and dataset, extending EEG decoding to executable structured programs. The use of component-level accuracy and full-program exact match enables more objective and fine-grained evaluation.
-
EEG signal-dependence validation. The authors include multiple control experiments, particularly matched EEG shuffling, to assess whether the model relies on genuine EEG signals. The substantial performance gap between real EEG and shuffled EEG provides evidence that trial-specific EEG information contributes to prediction performance.
Critical weaknesses
-
The paper does not clearly explain how schema-specific field names (e.g., region, sales) and filter values are generated, or whether the Data Table or Schema is available during inference. Key implementation details of the Teacher Representation, EEG Student architecture, alignment loss, and loss weighting are also insufficiently specified. Furthermore, the contributions of Grammar-constrained Decoding and the Schema Prior are not independently evaluated. Consequently, it is difficult to determine how much of the reported performance can be attributed to EEG decoding rather than external structural priors.
-
The paper does not verify whether the visualization mentally represented by participants during query interpretation matches the predefined ground truth. Additionally, the high chart-type accuracy achieved using only Duration and Padding raises concerns about potential dataset biases. The substantial performance gap between seen and unseen programs suggests limited compositional generalization. Finally, the large discrepancy between Render Rating ≥ 4 (73.3%) and Full Program Exact Match (27.33%) lacks sufficient explanation or execution-level validation. These issues limit the strength of the paper's claims about decoding structured visualization intent from EEG.
Questions for authors
-
How are schema-specific field names (e.g., region, sales) and filter values generated? Is the Data Table or Schema available during inference, or are these elements predicted directly from EEG? Could the authors also clarify the output spaces of the prediction heads, including the number of chart types and how clauses are represented?
-
Figure 5 suggests that the Qwen3 Teacher receives both the Data Table and Text Query. How is the teacher representation constructed (e.g., selected layers, pooling)? Could the authors clarify the EEG Transformer architecture, the definition of the alignment loss, and the selection of loss weights ( and )? An ablation without teacher alignment would help establish its contribution.
-
How are the valid program candidates and schema-conditioned prior in Equation (10) constructed? Could the authors report results for unconstrained decoding, grammar-constrained decoding, the full model, and Schema Prior Only? These comparisons are important for determining how much of the final performance is attributable to EEG rather than structural priors.
-
Did the experimental protocol verify whether participants' mentally represented visualizations matched the predefined ground truth? Without comprehension checks or intent verification, how can the authors distinguish decoding actual visualization intent from predicting stimulus-associated labels?
-
In Table 2, Duration alone achieves 51.54% chart-type accuracy, increasing to 58.73% with Padding. What classifiers are used for these baselines, and how can such high accuracy be explained? Furthermore, does the random split allow identical queries or visualization stimuli across training and test sets? Given the substantial performance drop on unseen programs (Table 4), could the authors clarify stimulus overlap and evaluate generalization under stricter splits?
-
Why does Render Rating ≥ 4 reach 73.3% while Full Program Exact Match is only 27.33%? Do these cases represent semantically equivalent programs, or merely visually similar charts? Execution-based equivalence results or representative examples would help clarify this discrepancy.
-
The title and abstract in the submitted PDF appear to differ from those on OpenReview.