- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose MindLens, an interface that assigns explicit semantic roles to soft tokens to verify which soft tokens generated from fMRI represent and which expressions language models actually use, and to check whether the tokens are used in actual sentence generation.
Critical strengths
[1] The authors show that semantic role tokens have a substantial influence on generating captions from brain signals.
[2] The authors evaluate in detail the impact of representations on information and output through various ablation experiments.
[3] The authors use readout analysis to evaluate properties of intermediate representations that captioning metrics such as BLEU, CIDEr, and ROUGE alone cannot reveal.
Critical weaknesses
[1] The authors have not sufficiently demonstrated the added value of MindLens over post-hoc alignment or its generalisability across encoders. Although 8 semantic role tokens improve direct readout and intervention selectivity, the paper does not compare MindLens with post-hoc alignment applied to the backbone, and the interpretability analyses are limited to the authors’ own backbone.
[2] Test captions were used to construct the concept vocabulary, and the descriptions in the main text and appendix are inconsistent. Section 2.4 states that concepts are extracted from training captions, whereas Appendix A.5 explicitly states that the procedure, including vocabulary selection and normalisation, was applied to captions from all 73,000 stimulus images without a train/test split.
Questions for authors
[1] Regarding Weakness 1, could the authors compare MindLens with post-hoc alignment applied to the backbone, with the alignment fitted on separate calibration data and evaluated on an independent test set?
[2] Regarding Weakness 2, could the authors clarify the actual data scope and procedure used to construct the concept bank and groups?
[3] COCO dataset provides 80 object categories with 12 super-categories. Some of the author's categories have identical or similar name to COCO super-categories, but their memberships differ. Could the authors (i) clarify the criteria for assigning concepts to groups beyond the requirement that each group have a coherent name, and (ii) explain why a custom caption-derived taxonomy was chosen instead of given coco categories?
[4] [minor] In Table 1, METEOR for Mind-SA, Yo’Mind, the backbone, and MindLens (38–40) is about twice that of the other methods (16–22), while CIDEr and SPICE show no comparable gap. Were all scores computed with the same METEOR implementation? Were the baseline numbers reproduced under the same pipeline, or copied from the original papers?
[5] [minor]Section 2.4 describes clustering concept embeddings with K-means followed by manual adjustment, whereas Appendix A.5 describes a manually constructed grouping and presents K-means partitions as ablations. Could the authors clarify the relationship between these procedures and explain how the final groups were constructed?
5 comments
[Q2 + W2] Data scope of the concept bank and groups
We agree that the vocabulary should not be built from test captions, and that Section 3.4 was inaccurate. Rebuilt without the test images, the vocabulary keeps the same 132 concepts and eight groups, and the results change only marginally.
Original scope. As Appendix A.7 states, vocabulary selection and normalization were run over the captions of all 73,000 NSD stimulus images, including the 1,000 shared test images. Section 3.4 wrongly said "training captions". The vocabulary is a frequency ranking over human captions and involves no model and no fMRI data. We apologize for this mistake.
Procedure. The full construction is given in our answer to Q5 below. The groups take only the concept list as input.
Rebuilt without the test images. We reran the same pipeline on the captions of the 72,000 non-test images and retrained MindLens on the resulting labels, with nothing else changed. The rebuilt vocabulary has the same 132 concepts, with the same names, order, and groups. Two member words change, both through the rank-200 cutoff: path falls below it and leaves concept sidewalk, and glasses rises above it and joins concept eyewear, as already assigned in the concept file. Labels change on 540 of the 73,000 images, 532 of which lose sidewalk.
Caption quality is essentially unchanged:
| Vocabulary | B@1 | B@2 | B@3 | B@4 | METEOR | ROUGE | CIDEr | SPICE |
|---|---|---|---|---|---|---|---|---|
| Paper (with test images) | 59.56 | 41.41 | 27.49 | 18.05 | 40.33 | 45.08 | 64.80 | 13.77 |
| Rebuilt (no test images) | 59.62 | 41.52 | 27.74 | 18.50 | 40.37 | 45.10 | 65.69 | 13.75 |
Readability and selectivity. Selectivity is identical (55/56), and readout and removal effects change only slightly (drops in recall points):
| Vocabulary | Probe-free mAP | Mean own-group drop | Mean spillover |
|---|---|---|---|
| Paper (with test images) | 0.410 | 6.52 | 1.35 |
| Rebuilt (no test images) | 0.412 | 6.89 | 1.23 |
Revision. We corrected Section 3.4 and added this comparison to the appendix. The final version reports all results with the rebuilt vocabulary.
[Q3] Assignment criteria and the choice of taxonomy
Our groups follow the roles concepts play in a scene description, not object types, because the channels feed caption generation and the causal test counts what generated captions mention. COCO's supercategories group object types instead, so shared names differ in meaning.
Assignment criteria
Beyond nameability, three rules decided the assignment:
- Starting point. Groups start from k-means clusters of the concept-name embeddings (Q5), so concepts the language model represents similarly start together.
- Scene role. Following the structure of scene descriptions (
Appendix A.7), the groups cover who is present (people), animals, food, the indoor or outdoor setting, vehicles, activities (sport), and other objects. - Context words join the objects they host. For example, zoo joins animals group; kitchen, counter, stove, refrigerator and restaurant join food group, and train track, train station, airport and luggage join vehicle group.
Each concept belongs to exactly one group, and groups are not balanced by size.
Why not COCO's categories
Captions, not object annotations. The causal test (Section 4.4) counts which concepts a generated caption mentions, so the concepts must be the words captions use. COCO's 80 categories are annotated object instances and do not cover scene context such as street, field, water, or grass. By our count, 51 of our 132 concepts are COCO category names, and none of the 26 outdoor concepts is. Yet the outdoor channel shows the largest own-group drop in the paper (14.61 points).
Matched to a 3-second view. NSD participants viewed each image for 3 seconds, enough for the gist of the scene but not every small object. COCO's annotators had ample time to mark every instance of the 80 categories, including small objects a participant may never register, and a caption can describe a scene without naming any annotated object. Captions name a scene's salient content, which is closer to what a participant takes in.
Same names, different meanings. COCO's "outdoor" is street furniture (traffic light, fire hydrant, stop sign, parking meter, bench), four of which are in our object group, and its "indoor" is small household items such as books, clocks, and vases. Our indoor and outdoor groups describe the setting.
The conclusions do not rest on our taxonomy. They hold across five partitions, including k-means partitions with no manual step (Table 10): probe-free mAP stays between 0.402 and 0.410, and selectivity between 0.789 and 0.967, against 0.464 for the backbone (chance 0.49).
Revision. We state these criteria in Appendix A.7 and add the comparison with COCO's categories.
[Q4] METEOR in Table 1
Thank you for pointing this out. We share this concern and noticed the same discrepancy when preparing Table 1.
To our knowledge, Yo'Mind (ICLR 2026) remains the state-of-the-art brain captioning model built on a lightweight language model (GPT-2). As a peer-reviewed work, we considered its reported results a reliable reference and followed it closely: nearly all baseline numbers in Table 1 are copied directly from the Yo'Mind paper.
We also attempted a quick re-evaluation of Yo'Mind, but only its code is released, without trained checkpoints. The same holds for Mind-SA, MindGPT and MindArt, which share the same first author and research group. Interestingly, MindGPT and MindArt report METEOR around 20, whereas Mind-SA and Yo'Mind report around 40, although their other metrics show no comparable gap. We are therefore also unsure whether the METEOR values in these papers were computed with the same implementation. What we can confirm is that our own METEOR scores are computed with NLTK 3.9.4. We will contact the authors of Yo'Mind to confirm their implementation before the final revision.
[Q5] How the final groups were constructed
The eight groups were obtained by running k-means and then adjusting the clusters by hand. Section 3.4 and Appendix A.7 each described only part of this procedure; we clarify it step by step below:
- Step 1: concepts. We extract candidate nouns from the stimulus captions, merge spelling variants and synonyms through a hand-authored lookup table (no embedding or clustering), and keep the 200 most frequent entries by image document frequency. These are collected into 132 concepts in a fixed file that assigns each surface form to exactly one concept (
Appendix A.7). The variation that Section 3.4 illustrates with dog, puppy and canine is handled here, not by clustering. - Step 2: groups. We ran k-means () over the frozen GPT-2 input embeddings of the 132 concept names to obtain initial clusters, and then adjusted them by hand so that every group carries a single natural name; a channel whose group cannot be named is not interpretable.
- Step 3: fixed before training. The concept file and the partition were fixed before any model was trained and are released with the code. Rebuilding the vocabulary without the test images (Q2) only changed which surface forms pass the rank cutoff: path fell below rank 200 and glasses rose into it, both already assigned in the file, so the 132 concepts and the eight groups are unchanged.
Where k-means enters. K-means enters twice: as the starting point of the hand-adjusted partition (Step 2 above), and on its own in Table 10, where partitions produced by k-means alone, without the manual adjustment, serve as an ablation at $N_c = $ 6, 8 and 10. These give the same readout (probe-free mAP 0.402 to 0.406, against 0.410) and selectivity far above the backbone (0.789 to 0.967, against 0.464; chance 0.49), so the conclusions do not depend on the manual adjustment. We keep the hand-adjusted groups because they can be named, and at they are also more selective (0.929 against 0.875).
Revision. We rewrote Section 3.4 to describe the two steps separately, added the k-means starting point to Appendix A.7, and updated Figure 1 accordingly, so that all three describe the same procedure.
We appreciate Reviewer Dbae for your insights and careful reading, which led to the new results detailed below. The five parts below follow your questions and name the weakness each one addresses. (W and Q stand for Weaknesses and Questions, respectively.)
[Q1 + W1] Comparison with post-hoc alignment on the backbone
Thank you for suggesting this comparison. A post-hoc alignment fitted on the backbone finds group directions, but removing them changes the captions only slightly: the mean own-group drop is at most 0.81 points, compared with 6.52 for MindLens. This suggests that aligning the channels during training is what makes them causally important for the captions.
Post-hoc alignment. We kept the paper's backbone unchanged. On the 36,000 training rows, the same rows and labels MindLens is trained with, we computed for each group and each of the 48 tokens the group's direction: the mean token over images containing the group minus the mean over images without it. Each group's post-hoc token is the one whose projection onto this direction best separates the group. The alignment is evaluated on the 4,000 test rows, none of which is used for fitting. Each group chooses its token freely, which gives the backbone more freedom than MindLens's one channel per group.
Two post-hoc edits. Both use the paper's removal test and the same fill value as for MindLens: the token's mean over test images lacking the group.
What this shows:
Section 4.3): in the backbone, the information is spread over the prefix.Section 4.3). The added value of MindLens is therefore not readability but channels that can be read without a fitted probe and are causally necessary.Generalization across encoders. This is a fair point. Our analyses have so far been carried out on our own fMRI encoder, and whether the findings extend to other encoder architectures remains an open question that we have not yet been able to test. In the revision, we will make this scope clear and note evaluation on other encoders as future work.