- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a reinforcement learning framework VPM, to encourage stimulus-faithful image captioning from brain activity. The proposed method combines a brain-conditioned captioning policy with a caption-conditioned flow model, using weighted cosine agreement between clean-feature estimates for generated and reference captions at shared image-derived noisy anchors as a reward to optimize the language readout through KL-regularized GRPO. Experiments on NSD and THINGS-EEG show improvements over supervised initialization across two VLM backbones.
Critical strengths
- The proposed method is well motivated by the need for stimulus-faithful brain image captioning. By highlighting how strong language priors can produce plausible but stimulus-inconsistent descriptions, the paper provides a clear rationale for using a visual posterior matching reward to guide reinforcement learning.
- Architecturally, the method coherently integrates a brain-conditioned captioning policy with a caption-conditioned flow critic. By comparing clean-feature estimates for generated and reference captions at shared image-derived noisy anchors, it constructs a structured visual reward for language-policy optimization.
- Experimental results demonstrate the effectiveness of the proposed method. The paper includes thorough comparisons with 9 existing methods across multiple standard metrics, demonstrating clear performance advantages on the NSD dataset.
Critical weaknesses
-
Although the paper specifies the adopted BrainXS encoder and alignment objective, I think insufficient implementation details on brain–visual alignment make it difficult to reproduce the mapping from fMRI and EEG inputs to the token spaces of the VLM backbones. In particular, clarifying how the neural inputs are represented and what output token shapes are used would improve the reproducibility of the initialization stage.
-
Related studies (Jing et al., Horikawa, Chen et al.) also explore generating descriptions of visual content from brain activity. I think discussion of these studies could help readers better understand how VPM differs in representation alignment, caption generation, and stimulus faithfulness.
[1]Jing et al., Beyond brain decoding: Visual-semantic reconstructions to mental creation extension based on fmri. ICCV, 2025 [2]Horikawa, Mind captioning: Evolving descriptive text of mental content from human brain activity. Science Advances, 2025 [3]Chen et al, Reducing semantic mismatch in brain-to-text decoding through personalized multimodal masking. ICLR, 2026.
Questions for authors
- How was the number of noisy anchors K=5 in Equation (9) selected? Could the authors report captioning performance and computational cost for different values of K?
- Could the authors also explain why the EEG evaluation in Table 2 was limited to subject 8 of the THINGS-EEG dataset and clarify whether the improvements hold across additional EEG subjects?
- How much of the observed performance gain comes from better use of stimulus-specific information in brain signals, and how much from captioning post-training itself? Are the existing experiments sufficient to disentangle these contributions?
- Could the authors provide paired confidence intervals or significance tests for the BLEU-4 and CIDEr improvements of VPM over the alternative post-training objectives in Table 3?
- For stimuli with similar compositions but different key object attributes, can VPM provide reliable rewards?
2 comments
Response to Reviewer Y9j4 (2/2)
Q2. EEG evaluation beyond subject 8
Our initial submission reported subject 8 for the EEG cross-modality comparison. We have now added all nine remaining THINGS-EEG subjects with both backbones. Revised Table 2 reports equal-weight means over all ten subjects, and Appendix C.2, Table 9 provides every subject’s results.
| Backbone | BLEU-4: Base → +VPM | CIDEr: Base → +VPM |
|---|---|---|
| SmolVLM2-2.2B | 3.75 → 4.37 | 15.31 → 16.70 |
| Qwen2.5-VL-3B | 4.49 → 4.93 | 17.68 → 18.41 |
The ten-subject means improve on all nine metrics for both backbones.
Q3. Stimulus-specific information versus captioning post-training
During RL, the brain encoder is frozen and VPM optimizes the language readout from its fixed neural representation. Table 3 compares alternative post-training objectives from the same initialized policy, supporting an additional benefit from the VPM objective.
To test whether VPM’s advantage over alternative post-training objectives depends on stimulus matching, we kept the DPO, CLIP-RL and VPM captions fixed and scored them against either the correct references or references randomly reassigned from other images. We used 20 random reassignments, shared across all methods and four subjects; shuffled gains are averaged across these reassignments.
| Comparison | BLEU-4 Δ: correct | BLEU-4 Δ: shuffled | CIDEr Δ: correct | CIDEr Δ: shuffled |
|---|---|---|---|---|
| VPM − DPO | +2.34 | +0.49 | +4.31 | +0.20 |
| VPM − CLIP-RL | +1.39 | +0.28 | +1.94 | +0.22 |
VPM’s gains over DPO and CLIP-RL become much smaller when the image–reference correspondence is broken. This supports a stimulus-associated benefit of VPM beyond a stimulus-independent change in wording.
Q4. Paired confidence intervals for Table 3
For the Table 3 comparisons, we computed paired 95% confidence intervals using 5,000 bootstrap resamples of the 982 test images, keeping the four subjects together. The table reports VPM (x-prediction) minus each alternative, using unrounded scores.
| Comparison | Δ BLEU-4 | 95% CI | Δ CIDEr | 95% CI |
|---|---|---|---|---|
| vs DPO | +2.337 | [1.633, 2.997] | +4.312 | [2.833, 5.817] |
| vs CLIP-RL | +1.395 | [0.937, 1.807] | +1.939 | [0.875, 2.958] |
| vs VPM (ODE) | +0.569 | [0.308, 0.798] | +0.645 | [0.141, 1.135] |
VPM improves both metrics over all three alternatives: all six confidence intervals are above zero. Paired randomization tests also remain significant after Holm correction (adjusted p < 0.05). These intervals reflect test-image uncertainty for the evaluated models.
Q5. Reward discrimination for similar scenes with different key details
We added a caption-swap analysis in Appendix C.8, Figure 8, using three selected NSD subject-1 image pairs: holding versus riding a surfboard, one versus two horses, and driving versus parked bus scenes. For each target, we compare its original VPM caption with the caption from the other image, holding the target image, reference caption, and anchor noise fixed. We use a frozen reproduced critic with K=5 and five noise seeds.
The matched caption receives higher reward in both directions for all three pairs, at every evaluated noise seed. For example, on the surfboard-holding image, “A man holding a surfboard on the beach” receives a mean reward of 1.892, versus 1.792 for “A man riding a surfboard on top of a wave.” On the single-horse image, the corresponding rewards are 1.844 for the single-horse caption and 1.769 for the two-horse caption. These examples illustrate that the reward can distinguish activity and number differences even when the captions share their principal object category.
References
[1] Jing et al. Beyond Brain Decoding: Visual-Semantic Reconstructions to Mental Creation Extension Based on fMRI. ICCV, 2025.
[2] Horikawa. Mind captioning: Evolving descriptive text of mental content from human brain activity. Science Advances, 2025.
[3] Chen et al. Reducing Semantic Mismatch in Brain-to-Text Decoding Through Personalized Multimodal Masking. ICLR, 2026.
Response to Reviewer Y9j4 (1/2)
We sincerely thank you for your careful review and constructive suggestions, and for recognizing the motivation and structured reward design of our work. We have highlighted the corresponding revisions in orange in the revised manuscript for clarity. Below, we respond to the two weaknesses and each of your five questions.
W1. Implementation details of brain–visual alignment
We have added the requested implementation details in Appendix C.4 (Brain–Visual Alignment Details), including the fMRI/EEG input representations, BrainXS mapping, backbone-specific token shapes, and initialization procedure. Please refer to this subsection for the full specifications.
W2. Relation to Jing et al., Horikawa, and Chen et al.
We have added the recommended studies [1–3] and a brief discussion in the revised Related Work, under Brain Decoding. These studies emphasize multimodal alignment [1], iterative text optimization against brain-decoded features [2], and personalized semantic masking [3], respectively. VPM focuses on a stimulus-anchored visual posterior matching reward for language-policy optimization after brain–visual alignment. Please refer to the revised subsection for the related approaches to representation alignment and semantic readout.
Q1. Choice of K and computational cost
We use K=5 to compare caption-conditioned visual estimates at several noise levels while keeping the critic computation small. To examine this choice, we now report the anchor-count and placement ablations in Section 4.4, Tables 6–7 of the main text. The following results are equal-weight means over NSD subjects 1, 2, 5, and 7 with Qwen2.5-VL-3B; slots are scheduler index fractions.
Performance varies only modestly across the tested anchor configurations, with K=5 yielding slightly higher BLEU-4 and CIDEr scores. With K=5 fixed, performance also varies little across the tested anchor windows.
The estimated critic costs (dense-matrix GFLOPs per caption) are:
At K=5, the critic requires 1.389 GFLOPs per caption, compared with approximately 400.9 GFLOPs for Qwen2.5-VL-3B caption generation under the same accounting. The critic is used only during training and adds no captioning inference cost.