acceptodds
Under review as a conference paper at ICLR 2027

Brain-IT-VQA: From Brain Signals to Answers

Abstract

Decoding visual content from fMRI signals recorded while a person views images, and specifically answering questions about the seen images, is a long-standing challenge. While significant progress has been made in recent years in visual question answering (VQA) from fMRI, performance remains limited. Moreover, although recent models can make increasingly accurate predictions, they have rarely been used as tools for understanding the structure of visual representations in the brain. We present **Brain-IT-VQA**, a framework for visual question answering from fMRI. Unlike previous methods, which extract a fixed representation from the fMRI signal, our extraction is conditioned on the question itself, so what is decoded from brain activity depends on what is being asked. Our model substantially outperforms previous fMRI-based captioning and VQA approaches. We further introduce **NSD-VQA**, a new dataset and benchmark for visual question answering from fMRI. Unlike existing image-fMRI VQA datasets, which typically provide only a few broad and weakly controlled questions per image, NSD-VQA provides on average 20 question-answer pairs per image across 20 controlled question categories that disentangle multiple levels of visual understanding. This enables more reliable and interpretable evaluation despite limited fMRI test data. Together, Brain-IT-VQA and NSD-VQA provide both a strong predictive framework and a tool for studying brain representations. Using this benchmark, we quantify which forms of visual and semantic information can be reliably decoded from fMRI responses to natural images. We further analyze the contributions of different brain regions across question types.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Decoding, fMRI Decoding, fMRI Visual Question Answering, fMRI Image Captioning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RybJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Brain-IT-VQA, a framework for answering questions about viewed images directly from fMRI signals. Unlike methods that extract fixed representations, it conditions the decoding of brain activity on the question. The authors also introduce NSD-VQA, a large-scale benchmark providing approximately 20 question–answer pairs per image across controlled categories covering different levels of visual understanding. The paper reports improved performance over previous fMRI-based captioning and VQA approaches. Beyond predictive performance, the framework and benchmark are used to investigate which visual and semantic information can be decoded from fMRI and how different brain regions contribute across question types.

Critical strengths

(1) Previous methods typically extract a fixed representation from fMRI signals and then use a language model to answer questions based on that representation. The proposed method conditions feature extraction on the question, enabling it to capture question-relevant information from fMRI signals. This can help preserve important information that might otherwise be lost during feature compression.

(2) The paper introduces NSD-VQA, a dataset covering multiple dimensions of visual understanding, including object recognition, attributes, spatial relations, counting, actions, and scene information. The dataset supports fine-grained evaluation by category, helping reveal models’ capabilities and limitations in decoding different types of visual and semantic information.

Critical weaknesses

(1) The dataset relies heavily on model-generated question–answer pairs. Each of the ten human evaluators reviewed only the question–answer pairs associated with 50 images. The proportion of manually verified data is relatively small, and agreement among the evaluators is not discussed in sufficient detail.

(2) The core components, namely functionally similar brain voxels and the Brain Interaction Transformer, are inherited from Brain-IT. The contribution therefore appears to lie primarily in adapting an existing architecture to fMRI-based visual question answering, with relatively limited methodological novelty.

Questions for authors

(1) The components based on functionally similar brain voxels and the Brain Interaction Transformer originate from Brain-IT, while the cross-transformer architecture draws on InstructBLIP. The work therefore appears to focus primarily on adapting these existing components to fMRI-based visual question answering. The authors should clarify which architectural designs specifically address challenges associated with fMRI decoding.

(2) The paper does not report the final number of question–answer pairs. Instead, it reports only 73,000 images with corresponding fMRI recordings from eight subjects, along with approximately 123,000 unlabeled COCO images. A table comparing the proposed dataset with existing datasets should be provided to clarify its advantages.

(3) The 73,000 NSD images alone imply at least 1.46 million question–answer pairs. Given this scale, are 2,500 human evaluations involving 50 images and ten evaluators sufficient to provide a representative assessment of annotation quality? Were annotations corrected for samples where human judgments disagreed with the automatically generated annotations?

(4) The authors should report the professional backgrounds and relevant experience of the ten human evaluators and expand the manual assessment to cover more samples. Inter-rater agreement and agreement between human judgments and automatic annotations should be reported separately to assess the reliability of the generated annotations.

(5) The generation prompts in the appendix prioritize prominent, clearly visible objects and instruct the model to ignore some background or ambiguous objects. They also specify choosing the lower count when the number of objects is uncertain. Could these instructions introduce a systematic bias toward underestimating object counts?

(6) The ablation study in Table 5 supports the effectiveness of the overall Q-BIT design. However, Appendix B.1 indicates that Q-BIT also increases the hidden dimension and adds six attention modules relative to the original BIT. Did the BIT baseline use the same network architecture and training settings as Q-BIT? Could the authors add a control that retains the complete Q-BIT architecture but withholds the actual question? This would help isolate the additional performance gains attributable to conditioning fMRI feature extraction on the actual question.

7 comments

We thank the reviewer for their time, effort, and constructive feedback. We would like to emphasize that our primary contribution is a new architecture and methodology for fMRI-based visual question answering: to our knowledge, the first brain-to-language decoding approach in which fMRI processing itself is conditioned on the question, enabling task-dependent extraction of information from brain activity. This leads to substantial improvements over existing approaches, achieving SOTA results on established image captioning and VQA benchmarks, independent of our proposed dataset. We additionally introduce NSD-VQA, an fMRI-tailored benchmark enabling systematic evaluation of decodable visual and semantic information across 20 controlled categories. We conducted additional experiments and analyses in response to the reviewers' suggestions and address their comments below.

W1 : Manual data verification and evaluator agreement: ... Each of the ten human evaluators reviewed only the question–answer pairs associated with 50 images. The proportion of manually verified data is relatively small, and agreement among the evaluators is not discussed in sufficient detail.

The human evaluation was designed to assess annotation quality through random sampling with balanced coverage across question categories, rather than to manually verify a large proportion of the dataset. For a dataset of this scale, manually inspecting every QA pair is impractical, and the relevant consideration is whether the sampled evaluation provides a reliable estimate of annotation quality.

Ten independent annotators each evaluated five questions for 50 randomly sampled images. Each annotator received a different set of images. We therefore report human–VLM agreement separately for each annotator, together with aggregate statistics across annotators, to assess the consistency of the human evaluation.

The results show high and consistent human–VLM agreement, with a mean of 89.85% ± 2.16 (mean ± SD) across annotators and an average N/A response rate of N/A 6.07%. . These results indicate that agreement is consistently high across independent annotators rather than being driven by a particular participant. Individual annotator results are reported below.

Table R1. Per-annotator agreement with the generated annotations. Agreement and disagreement rates exclude N/A responses.

AnnotatorAgreementDisagreementN/A
A1197 (86.78%)30 (13.22%)9 (3.81%)
A2195 (88.24%)26 (11.76%)14 (5.96%)
A3198 (90.41%)21 (9.59%)16 (6.81%)
A4206 (92.79%)16 (7.21%)15 (6.33%)
A5186 (86.92%)28 (13.08%)21 (8.94%)
A6204 (90.27%)22 (9.73%)12 (5.04%)
A7200 (89.69%)23 (10.31%)13 (5.51%)
A8195 (89.04%)24 (10.96%)16 (6.81%)
A9211 (92.95%)16 (7.05%)11 (4.62%)
A10199 (91.28%)19 (8.72%)16 (6.84%)
Mean ± SD89.85 ± 2.16%10.16 ± 2.16%6.07 ± 1.44%

Additional statistical analyses, including overall and category-specific image-bootstrap confidence intervals, are provided in our response to Q3 and Appendix C.1 of the revised manuscript.

W2 : Limited methodological novelty: The core components, namely functionally similar brain voxels and the Brain Interaction Transformer, are inherited from Brain-IT. The contribution therefore appears to lie primarily in adapting an existing architecture to fMRI-based visual question answering, with relatively limited methodological novelty.

Functional voxel clusters and the Brain Interaction Transformer (BIT) do come from Brain-IT. However, Brain-IT reconstructs a single image from the fMRI signal, so its output is fixed and independent of any query. Our contribution is a new architecture for fMRI-based VQA. To our knowledge, it is the first brain-to-language decoding method in which the processing of the fMRI signal itself is conditioned on the question. Q-BIT integrates question tokens into the cross-transformer, so the question guides the extraction of task-relevant information directly from the Brain Tokens, and different questions extract different information from the same fMRI response. This is a fundamental change to how information is decoded from brain activity, not simply an adaptation of an existing architecture to a new output task. It builds on BIT and the instruction-aware Q-Former of InstructBLIP, and is trained with a new two-stage procedure.

These novel elements lead to significant gains. Many works have addressed brain captioning and VQA, with gradual progress: MindLLM, the strongest prior method, improved over the previous best by 0.91 points on VQA-v2 and 2.03 on FSVQA. Our approach improves over MindLLM by 5.47 and 9.00 points, respectively (Table 2), and outperforms all prior methods on every captioning and VQA metric, most by a wide margin. These gains demonstrate the strength of our novel approach.

Q1: Specification of architectural designs addressing fmri decoding challenges: ... The work therefore appears to focus primarily on adapting these existing components to fMRI-based visual question answering. The authors should clarify which architectural designs specifically address challenges associated with fMRI decoding.

Beyond the novelty discussed in W2, the following designs address challenges specific to fMRI decoding:

  • Question-conditioned extraction. fMRI is noisy, and compressing it into one fixed representation loses information. As the reviewer notes in Strength (1), conditioning the extraction on the question helps preserve question-relevant information. It improves accuracy by 1.54 points on VQA-v2 and 6.17 on FSVQA (Table 5), and the parameter-matched ablation confirms that this gain does not come from added capacity (see Q6).

  • Two-stage training. fMRI signals have no direct correspondence with the input space of a language model. Stage 1 trains Q-BIT to reproduce InstructBLIP's question-conditioned representations from fMRI alone. This aligns the brain-derived representation with what the frozen LLM expects before any language supervision.

  • Training on predicted fMRI. fMRI data per subject is limited. We train on predicted fMRI responses for COCO images that have no fMRI recordings. This improves accuracy by 2.47 points on VQA-v2 and 4.07 on FSVQA, with consistent gains across three different image-to-fMRI encoders (Tables S8, S9).

  • Functional voxel clusters shared across subjects (from Brain-IT). These handle cross-subject variability and let a single model learn from all subjects' data.

Q2 : Report final QA-Pairs number and Comparison to existing datasets: The paper does not report the final number of question–answer pairs. ... A table comparing the proposed dataset with existing datasets should be provided to clarify its advantages.

The final NSD-VQA dataset contains 1,080,457 QA pairs associated with 70,566 NSD images with corresponding real fMRI recordings. We have added the exact dataset statistics and a comparison with existing VQA datasets.

Importantly, previous datasets like VQA-v2 and FSVQA are general-purpose computer vision benchmarks rather than datasets specifically designed for fMRI-based visual question answering. Both contain questions about COCO images, some of which overlap with the NSD stimuli and can therefore be associated with NSD fMRI recordings. However, these datasets were not designed to systematically probe different types of visual information represented in neural signals.

In contrast, NSD-VQA provides over one million natural-language QA pairs associated with real NSD fMRI recordings, spanning 20 controlled semantic categories covering complementary aspects of visual understanding, including object identity, attributes, counting, actions, interactions, spatial properties, and scene-level information. Furthermore, its annotations are constructed through a structured generation pipeline incorporating cross-model verification and independent human evaluation. This combination of scale, controlled question design, and annotation quality assessment distinguishes NSD-VQA from existing VQA datasets.

For a direct comparison, we report the number of images and QA pairs in VQA-v2 and FSVQA restricted to images overlapping with NSD. The comparison is summarized below and has been added to Appendix C.2 of the revised manuscript.

Table R2. Comparison of NSD-VQA with existing VQA datasets. VQA-v2 and FSVQA counts are restricted to images overlapping with NSD.

DatasetImage sourceImagesQA pairsfMRI-specific
NSD-VQA (ours)NSD70,5661,080,457Yes
NSD-VQA (ours)COCO (external)123,1781,887,222Yes (predicted)
VQA-v2COCO70,566378,537No
FSVQACOCO70,566211,698No

Q3 : Representative human evaluations: ... Given this scale, are 2,500 human evaluations involving 50 images and ten evaluators sufficient to provide a representative assessment of annotation quality? Were annotations corrected for samples where human judgments disagreed with the automatically generated annotations?

The human evaluation provides a representative assessment of annotation quality through random sampling across images and balanced coverage of question categories. Importantly, the reliability of an estimated agreement rate is reflected in the precision of the estimate, rather than simply in the proportion of the full dataset that is manually inspected. To quantify this precision, we performed an image-level bootstrap with 10,000 resamples, keeping all judgments associated with each image together to account for potential dependencies between questions about the same image.. The resulting 95% confidence interval was [88.56%, 91.09%]. The relatively narrow interval indicates that the estimate is precise, with agreement remaining consistently high across bootstrap resamples. We additionally report confidence intervals separately for each question category, providing a measure of the uncertainty associated with each category:

Table R3. Per-category human agreement: 95% confidence intervals from an image-level bootstrap with 10,000 resamples.

Question category95% image-bootstrap confidence interval (%)
Action[79.09, 91.89]
Animal[92.13, 100.00]
Color[69.20, 79.49]
Counting[83.85, 90.24]
Food[87.50, 100.00]
Holding[85.45, 98.31]
Location[86.40, 92.64]
Scene[95.72, 98.97]
Vehicle[82.76, 98.18]
Yes/No[92.48, 96.77]

Regarding annotation corrections, cases with substantial discrepancies identified during the annotation and verification process were excluded from the final dataset. In addition, our pipeline includes independent cross-model verification of object presence and counting using Qwen3-VL-8B and Gemma-4-31B-it. Counting annotations are retained only when both models agree; cases of disagreement are discarded. This provides an additional quality-control mechanism complementing the independent human evaluation.

Q4 : Professional background of evaluators, human vs. automation agreement: The authors should report the professional backgrounds and relevant experience of the ten human evaluators and expand the manual assessment to cover more samples. Inter-rater agreement and agreement between human judgments and automatic annotations should be reported separately to assess the reliability of the generated annotations.

The manual evaluation task involves judging visually observable properties of natural-scene images, such as the presence of objects, activities, and high-level attributes, and does not require specialized professional or neuroscience expertise.

The evaluators were recruited on a voluntary basis from students at our institute who were not involved in this project, and they were unaware of how the annotations had been generated. All ten evaluators had completed at least the equivalent of a high school education and had normal or corrected-to-normal vision. For open-ended questions, evaluators selected the correct answer from four multiple-choice options, with distractors sampled from answers to questions of the same category for other images.

Regarding inter-rater agreement, as explained in our response to W1, annotators evaluated different image sets. Additional details on annotator backgrounds, human–VLM agreement, and overall and category-specific bootstrap analyses have been included in Appendix C.1.

Q5 : Potential bias in object counts due to prompt instruction:
The generation prompts in the appendix prioritize prominent, clearly visible objects and instruct the model to ignore some background or ambiguous objects. They also specify choosing the lower count when the number of objects is uncertain. Could these instructions introduce a systematic bias toward underestimating object counts?

The counting instructions are deliberately conservative, focusing on clearly visible and salient objects rather than exhaustive enumeration of background or partially occluded instances. This is consistent with our objective of evaluating visual information that can be reliably associated with the fMRI response. Although this criterion may favor lower counts in ambiguous cases, it avoids relying on uncertain visual estimates. Furthermore, object counts are independently estimated by two vision-language models (Qwen3-VL-8B and Gemma-4-31B-it), and annotations are retained only when both models agree. The human evaluation also shows high agreement for counting questions (87.14%), supporting the reliability of this category.

Reviewer_61MSOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Brain-IT-VQA, a state-of-the-art framework for visual captioning and VQA from fMRI brain recordings, which outperforms previous methods by a large margin. Unlike previous approaches, the extraction of information from the brain signal is conditioned on the question itself, so that different questions draw different information from the same recording.

Critical strengths

The paper presents Brain-IT-VQA, a state-of-the-art framework for visual captioning and visual question answering (VQA) from fMRI brain recordings, demonstrating substantial performance improvements over previous methods. The paper is well organized, with a clear overall structure and logical flow. The motivation and key contributions are articulated clearly, and the proposed approach is described in sufficient detail, including the model architecture, algorithmic design, and training procedures. A notable strength of the framework is its question-conditioned extraction of information from brain signals, which enables the model to retrieve different semantic information from the same fMRI recording in response to different questions. This is an innovative and promising approach to brain-based visual understanding. The experimental evaluation is comprehensive, with sufficient validation and comparisons against existing methods to demonstrate the effectiveness and advantages of the proposed framework. Overall, the paper presents a well-motivated and technically sound contribution, with promising potential for advancing fMRI-based visual understanding and brain–AI interaction.

Critical weaknesses

1)The topic is very interesting, using cross-modal learning to extract semantic information from fMRI images. The NSD dataset contains a relatively limited number of fMRI recordings, leaving room for further expansion. The NSD dataset (Allen et al., 2022) comprises 73K images with corresponding fMRI brain recordings from 8 subjects, as well as ∼123K unlabeled COCO images (Lin et al., 2014). 2)The paper proposes training a VQA model using images and QA pairs, which then serves as a teacher to supervise another VQA model based on fMRI recordings. This is a creative and interesting idea. However, it is unclear whether the alignment between images in the COCO dataset and the corresponding fMRI recordings is supported by clinical data collection records. For example, did the participants who provided the fMRI recordings view the corresponding COCO images while their brain activity was being recorded? Was there a rigorous and well-documented alignment between the images and the fMRI recordings? This information is crucial for the subsequent experimental validation of the proposed method. I recommend that the authors provide relevant evidence to substantiate the data alignment procedure. 3)The reference formatting should be standardized throughout the paper. It is recommended to complete missing bibliographic information, such as volume numbers and page numbers. It is recommended that the authors reorganize and standardize the references throughout the paper.

Questions for authors
  1. Is semantic alignment between COCO images and fMRI data necessary during fMRI data acquisition? If so, how is this alignment established during the experimental data collection process?
  2. Brain signal data acquisition typically involves clinical procedures. Could the authors provide documentation demonstrating the relevant qualifications and ethical approvals for fMRI data acquisition?
  3. Natural Scenes Dataset (NSD) is a large-scale 7-Tesla fMRI dataset recording brain activity of 8 subjects. The dataset includes 8 subjects, but Tables 1 and 2 report experimental results only for Subject 1. Why are the results for all subjects not presented?
  4. Both the data collection and model training procedures in this paper use the semantic information of COCO images as ground truth to train the model to recover image-aligned semantic information from fMRI recordings. As a topic for further discussion, brain activity elicited by viewing an image may encode information that extends far beyond the explicit semantic content of the image itself. Therefore, directly using image semantics as ground truth to constrain model training may impose certain limitations. Would it be more valuable to incorporate neuroscientific interpretations of fMRI signals to more accurately uncover and decode the rich semantic information reflected in brain activity?
4 comments

We thank the reviewer for their time, effort, and constructive feedback. We would like to clarify that our work does not involve collecting new fMRI data. Instead, we leverage the publicly available Natural Scenes Dataset (NSD), one of the largest and most comprehensive available resources for natural-image fMRI decoding, whose acquisition procedures, experimental protocols, and ethical approvals are documented in the original study (Allen et al., 2022). Our contribution focuses on developing a new methodology for question-conditioned decoding of visual and semantic information from existing fMRI recordings, while extending model training through additional external images with predicted fMRI responses. We address the reviewer's concerns below.

W1 : NSD dataset: ... The NSD dataset contains a relatively limited number of fMRI recordings, leaving room for further expansion. ...

The data acquisition was conducted as part of the original NSD study, rather than this work. NSD is exceptionally large for the field and represents one of the largest available datasets for natural-image fMRI decoding, comprising approximately 73K natural-scene images across eight participants, collected over 30–40 scanning sessions per participant. Substantially expanding such datasets requires considerable scanning time and resources. Our framework therefore focuses on making efficient use of the available recordings and supplements them through data augmentation with additional COCO images and predicted fMRI responses, as described in Section 3.3.

W2 : Correspondence between COCO images and fMRI recordings: ... did the participants who provided the fMRI recordings view the corresponding COCO images while their brain activity was being recorded? Was there a rigorous and well-documented alignment between the images and the fMRI recordings? ...

The data acquisition was conducted as part of the original NSD study, rather than this work. The original publication describes extensive testing and verification of the acquisition procedures, including subject selection to identify participants with strong stimulus-related responses (Allen et al., 2022).

W3: References: The reference formatting should be standardized throughout the paper. It is recommended to complete missing bibliographic information, such as volume numbers and page numbers. …

We have reviewed and fixed the bibliography for consistency with such details as volume, issue, and page number. If there are any additional issues let us know and we will address them.

Q1. Is semantic alignment between COCO images and fMRI data necessary during fMRI data acquisition? If so, how is this alignment established during the experimental data collection process?

Data acquisition was conducted as part of the original NSD study, rather than this work (Allen et al., 2022). Alignment between the COCO images and fMRI data was established by pairing each presented image with its corresponding recorded fMRI response. As part of the acquisition procedure, participants were instructed to fixate at the center of the images, and measures were taken to ensure that they paid attention to the stimuli.

Q2. Brain signal data acquisition typically involves clinical procedures. Could the authors provide documentation demonstrating the relevant qualifications and ethical approvals for fMRI data acquisition?

The fMRI data acquisition was conducted as part of the original NSD study, rather than this work. Allen et al. (2022) report that informed written consent was obtained from all participants and that the experimental protocol was approved by the University of Minnesota Institutional Review Board. The original publication documents the acquisition procedures and ethical approval. Our work uses this existing dataset and does not involve any additional fMRI data acquisition. Our human evaluation consists of a separate, non-medical annotation survey and does not involve collecting brain signals.

Q3. Natural Scenes Dataset (NSD) is a large-scale 7-Tesla fMRI dataset recording brain activity of 8 subjects. The dataset includes 8 subjects, but Tables 1 and 2 report experimental results only for Subject 1. Why are the results for all subjects not presented?

As stated in Section 5.1, we report results on Subject 1 in Tables 1 and 2 to follow the reporting convention of prior work and enable direct comparison under the same evaluation setting. Studies using NSD commonly evaluate Subjects 1, 2, 5, and 7, and we follow this practice. Results for all four evaluated subjects are provided in Appendix D.1, Table S2 for captioning, and Appendix D.2, Table S3 for VQA.

Q4. Both the data collection and model training procedures in this paper use the semantic information of COCO images as ground truth to train the model to recover image-aligned semantic information from fMRI recordings. As a topic for further discussion, brain activity elicited by viewing an image may encode information that extends far beyond the explicit semantic content of the image itself. Therefore, directly using image semantics as ground truth to constrain model training may impose certain limitations. Would it be more valuable to incorporate neuroscientific interpretations of fMRI signals to more accurately uncover and decode the rich semantic information reflected in brain activity?

We agree that image semantics do not capture the full information reflected in brain activity, which may also include subjective associations, memories, and internal states. Our use of image-based supervision therefore limits decoding to stimulus-related information rather than the complete semantic content of the recorded activity. Incorporating neuroscientific interpretations is a valuable future direction, although obtaining comprehensive, subject-specific semantic annotations from voxel-level activity remains challenging. Future fMRI foundation models may help support richer interpretations beyond the semantic content of the presented images

Dear Reviewer 61MS,

Can you please take a look at the authors' response and provide your further input? Thank you.

AC.

Thank you for the response. I am particularly concerned about the dataset collection and annotation, as these are critical factors that directly determine the value of the proposed algorithm. The authors' explanations have largely avoided addressing these issues. My concerns remain unresolved. After carefully reviewing all the comments and responses, I have decided to lower my rating.

Reviewer_8kfdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Brain-IT-VQA, a framework for decoding natural-language answers directly from fMRI responses to visual stimuli. The main technical contribution is a question-conditioned Brain Interaction Transformer (Q-BIT), which uses textual queries to guide cross-attention over brain tokens before projecting the resulting representations into the input space of a frozen language model. The model is trained through a two-stage procedure involving representation distillation from InstructBLIP and subsequent language-supervised fine-tuning. The authors also introduce NSD-VQA, a benchmark containing structured question-answer pairs across different visual and semantic categories, and use it to investigate decoding performance and regional contributions.

Critical strengths
  • The paper reports consistent improvements over existing brain captioning and VQA approaches.
  • NSD-VQA provides a more structured way to evaluate different aspects of visual information recoverable from fMRI.
Critical weaknesses
  • The central idea of conditioning feature extraction on a textual question closely follows the instruction-aware Q-Former design in InstructBLIP. More importantly, the current ablation does not fully isolate the benefit of question conditioning. Replacing Q-BIT with the original BIT architecture changes more than the availability of textual conditioning, since Q-BIT also introduces additional attention layers and architectural modifications. A more convincing comparison would retain the same backbone, parameter budget, and training procedure while disabling only the interaction between question tokens and brain tokens. This would directly test the principal methodological claim.

  • While the proposed Q-BIT introduces question conditioning during neural feature extraction, prior methods such as MindLLM already integrate fMRI representations with instruction tokens for task-dependent decoding. The main architectural distinction therefore lies in where the question interacts with the neural representations. Moreover, question-conditioned cross-attention closely follows the instruction-aware Q-Former in InstructBLIP. It remains unclear whether moving question conditioning into the brain encoder provides a substantial advantage over existing late-fusion approaches.

  • Although the annotation pipeline includes cross-model verification and human evaluation, the reported agreement is not uniformly high. For example, the human evaluation reports only 74.47% agreement for color questions, compared with substantially higher agreement for several other categories.

  • This is a solid study to fMRI-based language decoding. However, question-conditioned feature extraction is already well established in vision-language modeling, and the existing ablations do not fully establish the added value. Can the authors provide a parameter-matched ablation in which the question-to-brain cross-attention is disabled while the remaining Q-BIT architecture is preserved? This would help separate the contribution of question conditioning from the additional model capacity.

Questions for authors

Although the authors provide a question-only baseline, it would be useful to further assess the model's reliance on stimulus-specific neural information. How does performance change when fMRI responses are randomly mismatched with the corresponding questions and images at evaluation time? This control could help distinguish the contribution of stimulus-specific brain representations from question-dependent answer priors.

5 comments

We thank the reviewer for the thoughtful review. Our approach introduces novel elements: question-conditioned extraction from brain representations (Q-BIT), and the adaptation of the BIT architecture from Brain-IT to VQA through a new two-stage training procedure. We want to emphasize that these elements lead to significant gains. Many works have addressed brain captioning and VQA, with gradual progress: MindLLM, the strongest prior method, improved over the previous best by 0.91 points on VQA-v2 and 2.03 on FSVQA. Our approach improves over MindLLM by 5.47 and 9.00 points, respectively (Table 2), and outperforms all prior methods on every captioning and VQA metric, most by a wide margin. These gains demonstrate the strength of our ideas. The parameter-matched ablation the reviewer requested confirms that the gain from question conditioning does not come from added capacity (details in our response to W1).

W1+W4: Parameter Matched Ablation: ... the current ablation does not fully isolate the benefit of question conditioning. replacing Q-BIT with the original BIT architecture changes more than the availability of textual conditioning, since Q-BIT also introduces additional attention layers and architectural modifications. A more convincing comparison would retain the same backbone, parameter budget, and training procedure while disabling only the interaction between question tokens and brain tokens. ... We performed an ablation comparing our proposed question conditioning (Q-BIT) with the adapted BIT from Brain-IT (Table 5). Following the reviewer's request, we also ran parameter-matched ablations that keep the full Q-BIT architecture, as well as the LLM's access to the original question, but withhold the question from Q-BIT. Instead, Q-BIT receives either a fixed instruction ("Please describe the image briefly") or learned tokens. To also separate the effect of the training procedure, we crossed these with two stage-1 targets: the Q-Former captioning representation, as used for the BIT baseline, or the question-conditioned Q-Former representation, as used for Q-BIT.

As shown in Table R1 below, all parameter-matched variants without the question fall below Q-BIT on both VQA-v2 and FSVQA, and remain close to or below the smaller BIT baseline. The added layers and parameters therefore do not account for the gain. The gain comes from conditioning the fMRI processing on the question.

Table R1. Parameter-matched ablations of question conditioning (subject 1).

Q-BIT inputStage-1 targetVQA-v2 Acc.FSVQA Acc.
Fixed instructionCaptioning52.5646.33
Fixed instructionQuestion-conditioned51.1843.40
Learned tokensCaptioning55.3246.93
Learned tokensQuestion-conditioned53.2946.33
BIT (Table 5)Captioning56.0745.83
Question (Q-BIT)Question-conditioned57.6152.00

W2: Question interaction with neural representations: ... The main architectural distinction therefore lies in where the question interacts with the neural representations. ... It remains unclear whether moving question conditioning into the brain encoder provides a substantial advantage over existing late-fusion approaches.

Our contribution is a new architecture for fMRI-based VQA. To our knowledge, it is the first brain-to-language decoding method in which the processing of the fMRI signal itself is conditioned on the question, so what is extracted from brain activity depends on what is being asked. It builds on BIT and the instruction-aware Q-Former of InstructBLIP, and adapts them to brain signals. In MindLLM, the question interacts with a fixed fMRI representation during language decoding. In our method, the question guides what is extracted from the Brain Tokens before the representation reaches the LLM, so different questions extract different information from the same fMRI response. This is more than a difference in placement: it changes what neural information is extracted, not only how it is interpreted. This question-dependent extraction from structured brain representations is a central architectural contribution of our work, and it translates into substantial empirical improvements.

Our ablations test this directly (Table 5). In the BIT ablation and the new parameter-matched ablations, extraction is question-independent while the LLM still receives the question, which corresponds to late fusion. Question-conditioned extraction improves accuracy by 1.54 points on VQA-v2 and 6.17 points on FSVQA. The parameter-matched ablations confirm that this gain does not come from added capacity (see W1).

W3: Agreement of human evaluation across categories: ... the reported agreement is not uniformly high. For example, the human evaluation reports only 74.47% agreement for color questions, compared with substantially higher agreement for several other categories.

We agree that annotation agreement varies across question categories. The human evaluation yielded 89.85% overall agreement across the 2,500 evaluations conducted by ten independent participants. Although color questions show lower agreement (74.47%), the other nine evaluated categories all exceed 85%, with six exceeding 90%. Thus, the lower agreement is concentrated in one category rather than reflecting a general issue with annotation quality.

Color is particularly challenging in complex natural scenes, as determining the color of a specific object can be ambiguous due to lighting, shadows, variations in appearance, and the difficulty of associating a color with the correct object. These ambiguities can affect both vision-language models and human annotators.

More generally, creating reliable VQA annotations at scale is challenging, and widely used general-purpose VQA datasets also have limitations. For example, COCO-QA automatically generates questions from image captions, which can result in awkwardly phrased or grammatically incorrect questions and restricts question coverage to information explicitly mentioned in the captions. Moreover, its four question categories are highly imbalanced: approximately 70% of questions concern object recognition, while counting and location questions account for only 7.5% and 6.1%, respectively. NSD-VQA aims to mitigate some of these limitations through structured image annotations and template-based question generation across 20 controlled semantic categories, providing more consistent question formulation and broader coverage of visual properties relevant to fMRI-based VQA. Nevertheless, we acknowledge that some categories remain more challenging, and report category-specific agreement to make these differences explicit.

Q1: Stimulus specific neural information: ... How does performance change when fMRI responses are randomly mismatched with the corresponding questions and images at evaluation time? This control could help distinguish the contribution of stimulus-specific brain representations from question-dependent answer priors.

We performed an experiment evaluating decoding without providing any fMRI signal (see Table S6 in Appendix D.4). Following the reviewer’s question, we also added a random permutation test in which fMRI responses are mismatched with the corresponding stimuli at evaluation time, while keeping the questions and reference answers unchanged (Tables R2 and R3). Shuffled-fMRI results are averaged over 10 random shuffles.

With shuffled fMRI, accuracy drops far below the full model on both VQA-v2 and NSD-VQA, and on VQA-v2 even below the no-fMRI baseline (37.90 vs. 42.86). This shows that the model relies on stimulus-specific neural information rather than on question-dependent answer priors. We have added these results to the revised paper.

Table R2. NSD-VQA shuffled-fMRI baseline accuracy by category.

CategoryAcc (%)CategoryAcc (%)CategoryAcc (%)
action10.43electronic Y/N49.26person Y/N50.65
animal9.16food5.33scene57.82
animal Y/N44.09food Y/N54.91sport Y/N44.11
appliance Y/N58.20holding2.75street Y/N51.46
building Y/N52.17household Y/N50.05structure Y/N51.70
clothing Y/N53.16location5.93vehicle13.73
color32.44nature Y/N46.83vehicle Y/N50.09
counting42.87objects Y/N52.19

Table R3. VQA-v2 ablation results.

ConditionAccuracy (%)
Full model57.60
No fMRI42.86
Shuffled fMRI37.90

Dear Reviewer 8kfd,

Can you please take a look at the authors' response and provide your further input? Thank you.

AC.

Comments on the paper

1 comment

For clarity, we summarize the main points of our rebuttal:

Contribution. Our method introduces several key advances for fMRI-based VQA. The central one: the processing of the fMRI signal itself is conditioned on the question, so different questions extract different information from the same recording. To our knowledge, this is the first brain decoding method to do this. In prior methods, including MindLLM and Brain-IT (on which we build), the extraction from the fMRI signal is fixed and independent of the question.

Results. We improve over MindLLM, the previous strongest method, by 5.47 points on VQA-v2 and 9.00 on FSVQA. We outperform all prior methods on every captioning and VQA metric. For reference, MindLLM improved on its predecessor by 0.91 and 2.03 points. These gains demonstrate the strength of our novel approach. These results are on established benchmarks, independent of our proposed dataset.

Experiments added during the rebuttal.

  • Parameter-matched ablations, running Q-BIT without the question at equal capacity, to rule out that the gain comes from its extra layers. All such variants fall below the full model, indicating the gain comes from question conditioning.
  • Human evaluation statistics, addressing whether the human survey is sufficient to validate the automated annotation quality. It covers 500 distinct images and 2,500 QA pairs. Mean human-VLM agreement is 89.85%, consistent across all ten annotators, with a 95% bootstrap interval of [88.56%, 91.09%]. The narrow interval shows that the sample supports a precise estimate of annotation quality, and that this quality is high.

Data provenance. All fMRI data comes from the public NSD release. We collected no new brain data. Acquisition protocols and ethical approvals are documented in Allen et al. (2022).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.