- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Brain-IT-VQA, a framework for answering questions about viewed images directly from fMRI signals. Unlike methods that extract fixed representations, it conditions the decoding of brain activity on the question. The authors also introduce NSD-VQA, a large-scale benchmark providing approximately 20 question–answer pairs per image across controlled categories covering different levels of visual understanding. The paper reports improved performance over previous fMRI-based captioning and VQA approaches. Beyond predictive performance, the framework and benchmark are used to investigate which visual and semantic information can be decoded from fMRI and how different brain regions contribute across question types.
Critical strengths
(1) Previous methods typically extract a fixed representation from fMRI signals and then use a language model to answer questions based on that representation. The proposed method conditions feature extraction on the question, enabling it to capture question-relevant information from fMRI signals. This can help preserve important information that might otherwise be lost during feature compression.
(2) The paper introduces NSD-VQA, a dataset covering multiple dimensions of visual understanding, including object recognition, attributes, spatial relations, counting, actions, and scene information. The dataset supports fine-grained evaluation by category, helping reveal models’ capabilities and limitations in decoding different types of visual and semantic information.
Critical weaknesses
(1) The dataset relies heavily on model-generated question–answer pairs. Each of the ten human evaluators reviewed only the question–answer pairs associated with 50 images. The proportion of manually verified data is relatively small, and agreement among the evaluators is not discussed in sufficient detail.
(2) The core components, namely functionally similar brain voxels and the Brain Interaction Transformer, are inherited from Brain-IT. The contribution therefore appears to lie primarily in adapting an existing architecture to fMRI-based visual question answering, with relatively limited methodological novelty.
Questions for authors
(1) The components based on functionally similar brain voxels and the Brain Interaction Transformer originate from Brain-IT, while the cross-transformer architecture draws on InstructBLIP. The work therefore appears to focus primarily on adapting these existing components to fMRI-based visual question answering. The authors should clarify which architectural designs specifically address challenges associated with fMRI decoding.
(2) The paper does not report the final number of question–answer pairs. Instead, it reports only 73,000 images with corresponding fMRI recordings from eight subjects, along with approximately 123,000 unlabeled COCO images. A table comparing the proposed dataset with existing datasets should be provided to clarify its advantages.
(3) The 73,000 NSD images alone imply at least 1.46 million question–answer pairs. Given this scale, are 2,500 human evaluations involving 50 images and ten evaluators sufficient to provide a representative assessment of annotation quality? Were annotations corrected for samples where human judgments disagreed with the automatically generated annotations?
(4) The authors should report the professional backgrounds and relevant experience of the ten human evaluators and expand the manual assessment to cover more samples. Inter-rater agreement and agreement between human judgments and automatic annotations should be reported separately to assess the reliability of the generated annotations.
(5) The generation prompts in the appendix prioritize prominent, clearly visible objects and instruct the model to ignore some background or ambiguous objects. They also specify choosing the lower count when the number of objects is uncertain. Could these instructions introduce a systematic bias toward underestimating object counts?
(6) The ablation study in Table 5 supports the effectiveness of the overall Q-BIT design. However, Appendix B.1 indicates that Q-BIT also increases the hidden dimension and adds six attention modules relative to the original BIT. Did the BIT baseline use the same network architecture and training settings as Q-BIT? Could the authors add a control that retains the complete Q-BIT architecture but withholds the actual question? This would help isolate the additional performance gains attributable to conditioning fMRI feature extraction on the actual question.
7 comments
Q1: Specification of architectural designs addressing fmri decoding challenges: ... The work therefore appears to focus primarily on adapting these existing components to fMRI-based visual question answering. The authors should clarify which architectural designs specifically address challenges associated with fMRI decoding.
Beyond the novelty discussed in W2, the following designs address challenges specific to fMRI decoding:
-
Question-conditioned extraction. fMRI is noisy, and compressing it into one fixed representation loses information. As the reviewer notes in Strength (1), conditioning the extraction on the question helps preserve question-relevant information. It improves accuracy by 1.54 points on VQA-v2 and 6.17 on FSVQA (Table 5), and the parameter-matched ablation confirms that this gain does not come from added capacity (see Q6).
-
Two-stage training. fMRI signals have no direct correspondence with the input space of a language model. Stage 1 trains Q-BIT to reproduce InstructBLIP's question-conditioned representations from fMRI alone. This aligns the brain-derived representation with what the frozen LLM expects before any language supervision.
-
Training on predicted fMRI. fMRI data per subject is limited. We train on predicted fMRI responses for COCO images that have no fMRI recordings. This improves accuracy by 2.47 points on VQA-v2 and 4.07 on FSVQA, with consistent gains across three different image-to-fMRI encoders (Tables S8, S9).
-
Functional voxel clusters shared across subjects (from Brain-IT). These handle cross-subject variability and let a single model learn from all subjects' data.
Q2 : Report final QA-Pairs number and Comparison to existing datasets: The paper does not report the final number of question–answer pairs. ... A table comparing the proposed dataset with existing datasets should be provided to clarify its advantages.
The final NSD-VQA dataset contains 1,080,457 QA pairs associated with 70,566 NSD images with corresponding real fMRI recordings. We have added the exact dataset statistics and a comparison with existing VQA datasets.
Importantly, previous datasets like VQA-v2 and FSVQA are general-purpose computer vision benchmarks rather than datasets specifically designed for fMRI-based visual question answering. Both contain questions about COCO images, some of which overlap with the NSD stimuli and can therefore be associated with NSD fMRI recordings. However, these datasets were not designed to systematically probe different types of visual information represented in neural signals.
In contrast, NSD-VQA provides over one million natural-language QA pairs associated with real NSD fMRI recordings, spanning 20 controlled semantic categories covering complementary aspects of visual understanding, including object identity, attributes, counting, actions, interactions, spatial properties, and scene-level information. Furthermore, its annotations are constructed through a structured generation pipeline incorporating cross-model verification and independent human evaluation. This combination of scale, controlled question design, and annotation quality assessment distinguishes NSD-VQA from existing VQA datasets.
For a direct comparison, we report the number of images and QA pairs in VQA-v2 and FSVQA restricted to images overlapping with NSD. The comparison is summarized below and has been added to Appendix C.2 of the revised manuscript.
Table R2. Comparison of NSD-VQA with existing VQA datasets. VQA-v2 and FSVQA counts are restricted to images overlapping with NSD.
| Dataset | Image source | Images | QA pairs | fMRI-specific |
|---|---|---|---|---|
| NSD-VQA (ours) | NSD | 70,566 | 1,080,457 | Yes |
| NSD-VQA (ours) | COCO (external) | 123,178 | 1,887,222 | Yes (predicted) |
| VQA-v2 | COCO | 70,566 | 378,537 | No |
| FSVQA | COCO | 70,566 | 211,698 | No |
Q3 : Representative human evaluations: ... Given this scale, are 2,500 human evaluations involving 50 images and ten evaluators sufficient to provide a representative assessment of annotation quality? Were annotations corrected for samples where human judgments disagreed with the automatically generated annotations?
The human evaluation provides a representative assessment of annotation quality through random sampling across images and balanced coverage of question categories. Importantly, the reliability of an estimated agreement rate is reflected in the precision of the estimate, rather than simply in the proportion of the full dataset that is manually inspected. To quantify this precision, we performed an image-level bootstrap with 10,000 resamples, keeping all judgments associated with each image together to account for potential dependencies between questions about the same image.. The resulting 95% confidence interval was [88.56%, 91.09%]. The relatively narrow interval indicates that the estimate is precise, with agreement remaining consistently high across bootstrap resamples. We additionally report confidence intervals separately for each question category, providing a measure of the uncertainty associated with each category:
Table R3. Per-category human agreement: 95% confidence intervals from an image-level bootstrap with 10,000 resamples.
| Question category | 95% image-bootstrap confidence interval (%) |
|---|---|
| Action | [79.09, 91.89] |
| Animal | [92.13, 100.00] |
| Color | [69.20, 79.49] |
| Counting | [83.85, 90.24] |
| Food | [87.50, 100.00] |
| Holding | [85.45, 98.31] |
| Location | [86.40, 92.64] |
| Scene | [95.72, 98.97] |
| Vehicle | [82.76, 98.18] |
| Yes/No | [92.48, 96.77] |
Regarding annotation corrections, cases with substantial discrepancies identified during the annotation and verification process were excluded from the final dataset. In addition, our pipeline includes independent cross-model verification of object presence and counting using Qwen3-VL-8B and Gemma-4-31B-it. Counting annotations are retained only when both models agree; cases of disagreement are discarded. This provides an additional quality-control mechanism complementing the independent human evaluation.
Q4 : Professional background of evaluators, human vs. automation agreement: The authors should report the professional backgrounds and relevant experience of the ten human evaluators and expand the manual assessment to cover more samples. Inter-rater agreement and agreement between human judgments and automatic annotations should be reported separately to assess the reliability of the generated annotations.
The manual evaluation task involves judging visually observable properties of natural-scene images, such as the presence of objects, activities, and high-level attributes, and does not require specialized professional or neuroscience expertise.
The evaluators were recruited on a voluntary basis from students at our institute who were not involved in this project, and they were unaware of how the annotations had been generated. All ten evaluators had completed at least the equivalent of a high school education and had normal or corrected-to-normal vision. For open-ended questions, evaluators selected the correct answer from four multiple-choice options, with distractors sampled from answers to questions of the same category for other images.
Regarding inter-rater agreement, as explained in our response to W1, annotators evaluated different image sets. Additional details on annotator backgrounds, human–VLM agreement, and overall and category-specific bootstrap analyses have been included in Appendix C.1.
Q5 : Potential bias in object counts due to prompt instruction:
The generation prompts in the appendix prioritize prominent, clearly visible objects and instruct the model to ignore some background or ambiguous objects. They also specify choosing the lower count when the number of objects is uncertain. Could these instructions introduce a systematic bias toward underestimating object counts?
The counting instructions are deliberately conservative, focusing on clearly visible and salient objects rather than exhaustive enumeration of background or partially occluded instances. This is consistent with our objective of evaluating visual information that can be reliably associated with the fMRI response. Although this criterion may favor lower counts in ambiguous cases, it avoids relying on uncertain visual estimates. Furthermore, object counts are independently estimated by two vision-language models (Qwen3-VL-8B and Gemma-4-31B-it), and annotations are retained only when both models agree. The human evaluation also shows high agreement for counting questions (87.14%), supporting the reliability of this category.
We thank the reviewer for their time, effort, and constructive feedback. We would like to emphasize that our primary contribution is a new architecture and methodology for fMRI-based visual question answering: to our knowledge, the first brain-to-language decoding approach in which fMRI processing itself is conditioned on the question, enabling task-dependent extraction of information from brain activity. This leads to substantial improvements over existing approaches, achieving SOTA results on established image captioning and VQA benchmarks, independent of our proposed dataset. We additionally introduce NSD-VQA, an fMRI-tailored benchmark enabling systematic evaluation of decodable visual and semantic information across 20 controlled categories. We conducted additional experiments and analyses in response to the reviewers' suggestions and address their comments below.
W1 : Manual data verification and evaluator agreement: ... Each of the ten human evaluators reviewed only the question–answer pairs associated with 50 images. The proportion of manually verified data is relatively small, and agreement among the evaluators is not discussed in sufficient detail.
The human evaluation was designed to assess annotation quality through random sampling with balanced coverage across question categories, rather than to manually verify a large proportion of the dataset. For a dataset of this scale, manually inspecting every QA pair is impractical, and the relevant consideration is whether the sampled evaluation provides a reliable estimate of annotation quality.
Ten independent annotators each evaluated five questions for 50 randomly sampled images. Each annotator received a different set of images. We therefore report human–VLM agreement separately for each annotator, together with aggregate statistics across annotators, to assess the consistency of the human evaluation.
The results show high and consistent human–VLM agreement, with a mean of 89.85% ± 2.16 (mean ± SD) across annotators and an average N/A response rate of N/A 6.07%. . These results indicate that agreement is consistently high across independent annotators rather than being driven by a particular participant. Individual annotator results are reported below.
Table R1. Per-annotator agreement with the generated annotations. Agreement and disagreement rates exclude N/A responses.
Additional statistical analyses, including overall and category-specific image-bootstrap confidence intervals, are provided in our response to Q3 and Appendix C.1 of the revised manuscript.
W2 : Limited methodological novelty: The core components, namely functionally similar brain voxels and the Brain Interaction Transformer, are inherited from Brain-IT. The contribution therefore appears to lie primarily in adapting an existing architecture to fMRI-based visual question answering, with relatively limited methodological novelty.
Functional voxel clusters and the Brain Interaction Transformer (BIT) do come from Brain-IT. However, Brain-IT reconstructs a single image from the fMRI signal, so its output is fixed and independent of any query. Our contribution is a new architecture for fMRI-based VQA. To our knowledge, it is the first brain-to-language decoding method in which the processing of the fMRI signal itself is conditioned on the question. Q-BIT integrates question tokens into the cross-transformer, so the question guides the extraction of task-relevant information directly from the Brain Tokens, and different questions extract different information from the same fMRI response. This is a fundamental change to how information is decoded from brain activity, not simply an adaptation of an existing architecture to a new output task. It builds on BIT and the instruction-aware Q-Former of InstructBLIP, and is trained with a new two-stage procedure.
These novel elements lead to significant gains. Many works have addressed brain captioning and VQA, with gradual progress: MindLLM, the strongest prior method, improved over the previous best by 0.91 points on VQA-v2 and 2.03 on FSVQA. Our approach improves over MindLLM by 5.47 and 9.00 points, respectively (Table 2), and outperforms all prior methods on every captioning and VQA metric, most by a wide margin. These gains demonstrate the strength of our novel approach.