acceptodds
Desk rejected at ICLR 2027

Reinforcement Learning for Brain Image Captioning by Imposing Visual Posterior Matching

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Decoding visual semantics from brain activity can provide a natural-language account of perceived stimuli. While existing approaches mainly focus on image classification, retrieval, or reconstruction, brain image captioning provides a compositional readout of objects, attributes, and relations. However, powerful vision-language priors can turn incomplete and noisy neural evidence into plausible yet stimulus-inaccurate descriptions. We propose Visual Posterior Matching (VPM), a reinforcement learning framework for brain image captioning. VPM first aligns neural representations with image tokens to initialize a brain-conditioned language policy, then trains a conditional flow model on paired image and caption features. During reinforcement learning, VPM constructs shared noisy anchors from the paired stimulus and rewards agreement between the clean visual-feature estimates induced by generated and reference captions at the same anchors. This target-anchored reward is designed to keep generation tied to stimulus-specific evidence while allowing diverse phrasings. Experiments on fMRI and EEG show that VPM improves core captioning metrics over supervised initialization and outperforms existing brain-captioning systems on the evaluated NSD dataset.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Image Captioning, Brain-Computer Interface, EEG, fMRI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Y9j4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a reinforcement learning framework VPM, to encourage stimulus-faithful image captioning from brain activity. The proposed method combines a brain-conditioned captioning policy with a caption-conditioned flow model, using weighted cosine agreement between clean-feature estimates for generated and reference captions at shared image-derived noisy anchors as a reward to optimize the language readout through KL-regularized GRPO. Experiments on NSD and THINGS-EEG show improvements over supervised initialization across two VLM backbones.

Critical strengths
  1. The proposed method is well motivated by the need for stimulus-faithful brain image captioning. By highlighting how strong language priors can produce plausible but stimulus-inconsistent descriptions, the paper provides a clear rationale for using a visual posterior matching reward to guide reinforcement learning.
  2. Architecturally, the method coherently integrates a brain-conditioned captioning policy with a caption-conditioned flow critic. By comparing clean-feature estimates for generated and reference captions at shared image-derived noisy anchors, it constructs a structured visual reward for language-policy optimization.
  3. Experimental results demonstrate the effectiveness of the proposed method. The paper includes thorough comparisons with 9 existing methods across multiple standard metrics, demonstrating clear performance advantages on the NSD dataset.
Critical weaknesses
  1. Although the paper specifies the adopted BrainXS encoder and alignment objective, I think insufficient implementation details on brain–visual alignment make it difficult to reproduce the mapping from fMRI and EEG inputs to the token spaces of the VLM backbones. In particular, clarifying how the neural inputs are represented and what output token shapes are used would improve the reproducibility of the initialization stage.

  2. Related studies (Jing et al., Horikawa, Chen et al.) also explore generating descriptions of visual content from brain activity. I think discussion of these studies could help readers better understand how VPM differs in representation alignment, caption generation, and stimulus faithfulness.

[1]Jing et al., Beyond brain decoding: Visual-semantic reconstructions to mental creation extension based on fmri. ICCV, 2025 [2]Horikawa, Mind captioning: Evolving descriptive text of mental content from human brain activity. Science Advances, 2025 [3]Chen et al, Reducing semantic mismatch in brain-to-text decoding through personalized multimodal masking. ICLR, 2026.

Questions for authors
  1. How was the number of noisy anchors K=5 in Equation (9) selected? Could the authors report captioning performance and computational cost for different values of K?
  2. Could the authors also explain why the EEG evaluation in Table 2 was limited to subject 8 of the THINGS-EEG dataset and clarify whether the improvements hold across additional EEG subjects?
  3. How much of the observed performance gain comes from better use of stimulus-specific information in brain signals, and how much from captioning post-training itself? Are the existing experiments sufficient to disentangle these contributions?
  4. Could the authors provide paired confidence intervals or significance tests for the BLEU-4 and CIDEr improvements of VPM over the alternative post-training objectives in Table 3?
  5. For stimuli with similar compositions but different key object attributes, can VPM provide reliable rewards?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Y9j4 (1/2)

We sincerely thank you for your careful review and constructive suggestions, and for recognizing the motivation and structured reward design of our work. We have highlighted the corresponding revisions in orange in the revised manuscript for clarity. Below, we respond to the two weaknesses and each of your five questions.

W1. Implementation details of brain–visual alignment

We have added the requested implementation details in Appendix C.4 (Brain–Visual Alignment Details), including the fMRI/EEG input representations, BrainXS mapping, backbone-specific token shapes, and initialization procedure. Please refer to this subsection for the full specifications.

W2. Relation to Jing et al., Horikawa, and Chen et al.

We have added the recommended studies [1–3] and a brief discussion in the revised Related Work, under Brain Decoding. These studies emphasize multimodal alignment [1], iterative text optimization against brain-decoded features [2], and personalized semantic masking [3], respectively. VPM focuses on a stimulus-anchored visual posterior matching reward for language-policy optimization after brain–visual alignment. Please refer to the revised subsection for the related approaches to representation alignment and semantic readout.

Q1. Choice of K and computational cost

We use K=5 to compare caption-conditioned visual estimates at several noise levels while keeping the critic computation small. To examine this choice, we now report the anchor-count and placement ablations in Section 4.4, Tables 6–7 of the main text. The following results are equal-weight means over NSD subjects 1, 2, 5, and 7 with Qwen2.5-VL-3B; slots are scheduler index fractions.

KAnchor slotsBLEU-4CIDEr
10.123.8472.15
10.523.5171.60
30.1, 0.2, 0.323.6571.92
30.3, 0.4, 0.523.7372.07
50.1, 0.2, 0.3, 0.4, 0.524.3372.26

Performance varies only modestly across the tested anchor configurations, with K=5 yielding slightly higher BLEU-4 and CIDEr scores. With K=5 fixed, performance also varies little across the tested anchor windows.

The estimated critic costs (dense-matrix GFLOPs per caption) are:

Reward computationCritic GFLOPs per caption
x-prediction, K=10.278
x-prediction, K=30.834
x-prediction, K=51.389
ODE, 20 steps, one trajectory5.557

At K=5, the critic requires 1.389 GFLOPs per caption, compared with approximately 400.9 GFLOPs for Qwen2.5-VL-3B caption generation under the same accounting. The critic is used only during training and adds no captioning inference cost.

AuthorsOpenReview ↗

Response to Reviewer Y9j4 (2/2)

Q2. EEG evaluation beyond subject 8

Our initial submission reported subject 8 for the EEG cross-modality comparison. We have now added all nine remaining THINGS-EEG subjects with both backbones. Revised Table 2 reports equal-weight means over all ten subjects, and Appendix C.2, Table 9 provides every subject’s results.

BackboneBLEU-4: Base → +VPMCIDEr: Base → +VPM
SmolVLM2-2.2B3.75 → 4.3715.31 → 16.70
Qwen2.5-VL-3B4.49 → 4.9317.68 → 18.41

The ten-subject means improve on all nine metrics for both backbones.

Q3. Stimulus-specific information versus captioning post-training

During RL, the brain encoder is frozen and VPM optimizes the language readout from its fixed neural representation. Table 3 compares alternative post-training objectives from the same initialized policy, supporting an additional benefit from the VPM objective.

To test whether VPM’s advantage over alternative post-training objectives depends on stimulus matching, we kept the DPO, CLIP-RL and VPM captions fixed and scored them against either the correct references or references randomly reassigned from other images. We used 20 random reassignments, shared across all methods and four subjects; shuffled gains are averaged across these reassignments.

ComparisonBLEU-4 Δ: correctBLEU-4 Δ: shuffledCIDEr Δ: correctCIDEr Δ: shuffled
VPM − DPO+2.34+0.49+4.31+0.20
VPM − CLIP-RL+1.39+0.28+1.94+0.22

VPM’s gains over DPO and CLIP-RL become much smaller when the image–reference correspondence is broken. This supports a stimulus-associated benefit of VPM beyond a stimulus-independent change in wording.

Q4. Paired confidence intervals for Table 3

For the Table 3 comparisons, we computed paired 95% confidence intervals using 5,000 bootstrap resamples of the 982 test images, keeping the four subjects together. The table reports VPM (x-prediction) minus each alternative, using unrounded scores.

ComparisonΔ BLEU-495% CIΔ CIDEr95% CI
vs DPO+2.337[1.633, 2.997]+4.312[2.833, 5.817]
vs CLIP-RL+1.395[0.937, 1.807]+1.939[0.875, 2.958]
vs VPM (ODE)+0.569[0.308, 0.798]+0.645[0.141, 1.135]

VPM improves both metrics over all three alternatives: all six confidence intervals are above zero. Paired randomization tests also remain significant after Holm correction (adjusted p < 0.05). These intervals reflect test-image uncertainty for the evaluated models.

Q5. Reward discrimination for similar scenes with different key details

We added a caption-swap analysis in Appendix C.8, Figure 8, using three selected NSD subject-1 image pairs: holding versus riding a surfboard, one versus two horses, and driving versus parked bus scenes. For each target, we compare its original VPM caption with the caption from the other image, holding the target image, reference caption, and anchor noise fixed. We use a frozen reproduced critic with K=5 and five noise seeds.

The matched caption receives higher reward in both directions for all three pairs, at every evaluated noise seed. For example, on the surfboard-holding image, “A man holding a surfboard on the beach” receives a mean reward of 1.892, versus 1.792 for “A man riding a surfboard on top of a wave.” On the single-horse image, the corresponding rewards are 1.844 for the single-horse caption and 1.769 for the two-horse caption. These examples illustrate that the reward can distinguish activity and number differences even when the captions share their principal object category.

References

[1] Jing et al. Beyond Brain Decoding: Visual-Semantic Reconstructions to Mental Creation Extension Based on fMRI. ICCV, 2025.

[2] Horikawa. Mind captioning: Evolving descriptive text of mental content from human brain activity. Science Advances, 2025.

[3] Chen et al. Reducing Semantic Mismatch in Brain-to-Text Decoding Through Personalized Multimodal Masking. ICLR, 2026.

Reviewer_VoRcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of decoding (captioning) visual stimulus. The authors propose VPM, a reinforcement learning inspired method to finetune a VLM backbone for visual signal decoding. VPM uses a conditional-flow critic to guide captions toward the visual evidence of the paired stimulus. VPM yields consistent improvements over existing baselines.

Critical strengths
  • The approach appears innovative. The proposed VPM method improves captioning performance over vanilla RL and DPO.
  • The method seems simple and elegant.
Critical weaknesses
  • My main concern is w.r.t. experimental results.
    1. In Table 1, is the difference between Ours (Base) and the baselines mostly due to better pretrained VLM?
    2. In Table 2, is the goal here to show that VPM finetuning improves over base, which is supervised policy initialization? If so, the conclusion seems obvious. Any well tuned method to finetune the LLM decoder will yield superior captioning performance.
    3. I'm confused about why the four tables have slightly different dataset parameters. Table 1 is only on NSD subject 1, which is a much smaller dataset than Table 2, which contains 5 different subjects. Table 2 only has the Base-to-ours comparison. I think it would make more sense to have the full SOTA comparison as well as the comparison of VPM to other post-training objectives on the full available data.
Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Response to Reviewer VoRc (1/2)

We sincerely thank you for your careful review and constructive comments, and we have highlighted the corresponding revisions in blue in the revised manuscript for clarity.

W1. Contribution of the pretrained VLM

Our Base policy benefits from the pretrained VLM and the image-caption adaptation described in Section 4.1. Table 1 compares complete brain-captioning systems, whose backbones and training procedures differ; it therefore does not isolate the contribution of the pretrained VLM. We assess the additional benefit of VPM through comparisons starting from the same initialized policy, including the alternative post-training objectives in Table 3.

To examine whether this benefit also holds with a larger backbone, we additionally evaluated Qwen2.5-VL-7B on NSD subject 1. The 7B results are included in the revised Appendix C.3, Table 10. For reference, the table below also reproduces the submitted 3B Base and +VPM results from Table 1; training batch configurations differ between the two backbones:

Metric7B Base7B +VPM3B Base (Table 1)3B +VPM (Table 1)
BLEU-422.3323.7121.3724.82
CIDEr68.9172.0567.9974.00

VPM improves all nine reported metrics over the 7B Base policy. The 3B +VPM model in Table 1 reaches 24.82 BLEU-4 and 74.00 CIDEr, exceeding the 7B +VPM results above. These results show that VPM remains useful with the 7B backbone. They also suggest that a larger backbone does not necessarily yield better brain-captioning performance.

One possible explanation is that different pretrained models have distinct representational properties, and some representations may align more closely with brain signals. This interpretation is motivated by prior work using visual encoders with distinct inductive biases to align hierarchical visual embeddings with brain signals [1]. Conwell et al. [2] likewise found no significant model-size effect on visual brain predictivity in a controlled comparison of ViT-S/B/L models trained on the same dataset across SimCLR, CLIP, and SLIP objectives.

AuthorsOpenReview ↗

Response to Reviewer VoRc (2/2)

W2(a). What does Table 2 establish?

Table 2 evaluates the effect of VPM relative to supervised initialization across two backbones and two neural modalities, fMRI and EEG. Its purpose is to show how the gains vary across these settings. The revised table reports the change from Base for each metric.

W2(b). Are the gains simply due to post-training?

We agree that effective post-training can improve over supervised initialization. To assess the contribution of the VPM objective, Table 3 compares it with DPO and CLIP-RL, starting from the same initialized Qwen2.5-VL-3B policy and evaluating on the same four NSD subjects. VPM improves over both alternatives on all four reported metrics. Its BLEU-4/CIDEr gains are +1.39/+1.94 over CLIP-RL and +2.34/+4.31 over DPO. These results support an additional benefit of VPM relative to the tested post-training objectives.

W3. Why do the tables use different subject scopes?

Table 1 follows the commonly reported NSD subject-1 benchmark for comparability with prior work. The five participants in the submitted Table 2 comprise four NSD fMRI subjects and THINGS-EEG subject 8. Tables 3 and 5 already report means over all four evaluated NSD subjects (1, 2, 5, and 7), with a separate model trained for each subject. Following your suggestion, we have extended the system comparison to methods with available outputs for all four NSD subjects, as summarized below. Results are averaged across these subjects, with each subject evaluated on the same set of 982 test images. Section 4.2, Table 4 separately reports the subject-1 visual evaluation.

MetricSDReconBrainCapUMBRAEBrainDecOurs (+VPM)
BLEU-43.3014.0417.9118.8124.33
CIDEr13.5238.8655.0253.0672.26

For EEG, we have expanded the evaluation to all ten THINGS-EEG subjects using both VLM backbones: Qwen2.5-VL-3B and SmolVLM2-2.2B. Revised Table 2 reports ten-subject means, and Appendix C.2, Table 9 provides the per-subject results.

References

[1] Zheng et al. Learning Brain Representation with Hierarchical Visual Embeddings. ICLR, 2026.

[2] Conwell et al. A Large-Scale Examination of Inductive Biases Shaping High-Level Visual Representation in Brains and Machines. Nature Communications, 2024.

Reviewer_CvSdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses stimulus information loss in brain image captioning caused by language priors. Its conditional flow critic compares denoised visual estimates conditioned on generated and reference captions at shared noisy anchors from target images, rewarding agreement. Reinforcement learning freezes the brain encoder and critic while optimizing the language policy through KL-regularized GRPO. Across two backbones, four NSD subjects, and one THINGS-EEG subject, VPM improves BLEU-4 and CIDEr over supervised initialization, with comparisons and ablations examining design choices.

Critical strengths
  1. The paper is clearly written and easy to follow, with a smooth presentation of the research motivation and proposed method.

  2. The experimental evaluation is reasonably comprehensive

Critical weaknesses

The evaluation primarily relies on conventional captioning metrics, such as BLEU-4, CIDEr, and SPICE. It would be helpful to complement these metrics with an LLM-as-a-judge evaluation, ideally using a vision-capable model to assess generated captions against the corresponding images. This could provide additional insight into visual faithfulness, including whether the captions accurately describe salient objects, attributes, and relations while avoiding unsupported details

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer CvSd

We sincerely thank you for your careful review and constructive suggestion. We have highlighted the corresponding manuscript revisions in purple for clarity. Our original supplementary material included an image-grounded Qwen3-VL-8B evaluation, which provides initial evidence of improved visual faithfulness over Base. To address your suggestion more fully, we have now added Polos and a blinded GPT-6.1-Terra evaluation on all 982 subject-1 test images (Section 4.2, Table 4). We present these new results first and summarize the previously submitted Qwen3-VL-8B results afterward (Figure 6).

Image-grounded evaluation with GPT-6.1-Terra and Polos

To address your request for a vision-capable LLM judge, we use GPT-6.1-Terra to assess whether each caption correctly describes the objects, attributes and relations in its target image and covers the salient content. We evaluate all 982 NSD subject-1 test images and report Polos [1] as a complementary multimodal metric learned from human feedback. The full evaluation protocol is provided in Appendix C.6.

MetricSDReconOneLLMBrainCapBrainDECMEVOXUMBRAEOurs (Base)Ours (+VPM)
Polos [1]27.5424.3738.2138.3840.0146.8046.2948.37
GPT-6.1-Terra32.8421.4946.8743.3348.1457.6954.4858.15

VPM improves the Terra score from 54.48 for Base to 58.15 (+3.67), while Polos increases from 46.29 to 48.37. It also achieves the highest scores on both metrics among all eight methods. These image-grounded results support improved visual faithfulness over Base and the strongest overall caption quality among the compared methods. The visual-fidelity breakdown is reported in Appendix C.6, Table 13.

Supplementary evidence from the original submission

The image-grounded Qwen3-VL-8B evaluation [2] also favors VPM: the subject-1 judged-as-correct rate increases from 25.6% to 28.3%, with improvements in both creatures and objects groups (Figure 6; Appendix C.6).

References

[1] Wada et al. Polos: Multimodal Metric Learning from Human Feedback for Image Captioning. CVPR, 2024.

[2] Bai et al. Qwen3-VL Technical Report. arXiv:2511.21631, 2025.

Dear Reviewer CvSd,

Thank you again for your helpful suggestion. We have addressed your request for a vision-capable LLM judge by adding image-grounded GPT-6.1-Terra evaluation, complementary Polos scores, and visual-fidelity analyses on all 982 NSD subject-1 test images (Section 4.2, Table 4; Appendix C.6). VPM improves visual faithfulness over Base and achieves the highest mean Terra and Polos scores among the compared methods. As the discussion closes on Nov 18, we would be grateful to know whether you have any remaining concerns or further questions. Thank you for your time and consideration.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive feedback. Reviewer CvSd finds the paper “clearly written and easy to follow”; VoRc describes the method as “simple and elegant”; and Y9j4 considers it “well motivated” and highlights its structured visual reward. Their comments have helped us sharpen the contribution and strengthen the evaluation.

What this work studies. Brain image captioning seeks to describe a viewed image from the corresponding brain response. A brain response is only an incomplete and indirect observation of a visual stimulus. We study whether reinforcement learning can improve the fidelity of this brain-to-language readout.

Contributions

  • Stimulus-anchored policy optimization. VPM uses each visual stimulus to guide caption optimization from its paired brain response. During RL, the brain encoder remains fixed and only the language policy is updated, preserving the learned brain–visual interface.

  • A tractable visual posterior matching reward. At shared image-derived noisy anchors, a frozen conditional flow critic produces clean-feature estimates conditioned on generated and reference captions. Their weighted cosine agreement compares the captions' visual implications while allowing alternative wording. Direct x-prediction avoids full ODE integration, reducing critic computation while achieving better overall captioning performance (Table 3).

  • Empirical gains across objectives, backbones and neural modalities. Starting from the same initialized Qwen2.5-VL-3B policy, VPM outperforms CLIP-RL and DPO on the four-subject NSD evaluation (Table 3). The evaluation spans fMRI and EEG, two backbone families, and an additional 7B model, with image-grounded assessments supporting improved visual faithfulness.

Shared Questions

  • Does VPM improve visual faithfulness? We added GPT-6.1-Terra and Polos evaluation on all 982 NSD subject-1 test images. VPM improves Terra from 54.48 for Base to 58.15 (+3.67) and achieves the highest Terra and Polos means among the eight compared methods. The overall comparison is reported in Section 4.2, Table 4, with the visual-fidelity breakdown in Appendix C.6, Table 13.

  • Are the gains specific to VPM beyond generic post-training? Across four NSD subjects, VPM outperforms DPO and CLIP-RL from the same initialized policy on all four reported metrics (Table 3). Paired 95% confidence intervals for BLEU-4 and CIDEr gains are positive against both methods and the ODE variant. Reference-shuffling results further support stronger output–stimulus association (response to Y9j4, Q3–Q4).

  • Do the gains extend across subjects and backbones? EEG evaluation now covers all ten THINGS-EEG subjects with both backbones; the ten-subject means improve on all nine metrics (Table 2; Appendix C.2, Table 9). We also report a four-subject NSD system comparison and improvements over the 7B Base policy (response to VoRc; Appendix C.3, Table 10).

Revisions to the PDF

  • Purple (CvSd). Section 4.2, Table 4 and Appendix C.6, Table 13: GPT-6.1-Terra and Polos evaluation, the visual-fidelity breakdown, and the fixed evaluator prompt.
  • Blue (VoRc). Table 2 and Sections 4.1–4.2: revised EEG scope and ten-subject means. Appendix C.3, Table 10: the 7B-backbone comparison.
  • Orange (Y9j4). Introduction and Section 2.1: expanded EEG scope and related-work discussion. Appendix C.2, Table 9 and Appendix C.4: per-subject EEG results and alignment details. Appendix C.8, Figure 8 and Section 4.4, Tables 6–7: caption-swap and anchor analyses.

Detailed answers and additional statistical analyses are provided in the reviewer-specific replies below. Thank you again for your time and consideration.

AuthorsOpenReview ↗

Review and Reviewer-Author Discussion Summary (1/2)

Dear PCs, SACs, ACs, and Reviewers,

We thank the reviewers for their careful reading and constructive feedback. Their comments helped us clarify the reward design and broaden the evaluation.

Strength. We appreciate the reviewers'recognition of the following strengths.

  • A structured visual reward for stimulus-faithful captioning. The reviews describe VPM’s use of a conditional flow critic to tie caption optimization to the viewed image. Y9j4 highlights the shared image-derived anchors used to compare visual estimates for generated and reference captions (CvSd: Summary, VoRc: Summary, Y9j4: Strengths 1–2).
  • Broad evaluation and gains over baselines. CvSd noted the breadth of the evaluation. VoRc highlighted gains over DPO and the RL baseline, and Y9j4 highlighted the comparison with nine existing systems on NSD (CvSd: Strength 2, VoRc: Strength 1, Y9j4: Strength 3).
  • Clear presentation and simple design. CvSd found the paper clear and easy to follow, and VoRc described the method as “simple and elegant” (CvSd: Strength 1, VoRc: Strength 2).

Concerns and Our Addressing. Below we group the main concerns raised during review and discussion, and briefly summarize how we addressed them in the rebuttal and revised manuscript.

  • Concerns about experiment design.

    • (CvSd: Weakness) Image-grounded assessment of caption fidelity.

      Our Addressing. We added GPT-6.1-Terra and Polos on all 982 NSD subject-1 test images. From Base to VPM, Terra rises from 54.48 to 58.15 and Polos from 46.29 to 48.37, the highest means among eight methods (Section 4.2, Table 4). The Terra breakdown shows gains in object fidelity (62.02 to 66.40) and coverage (60.36 to 62.83). Appendix C.6, Table 13 and the evaluator prompt document the rubric and results.

    • (VoRc: Weakness 1) Contribution of the pretrained VLM.

      Our Addressing. Table 1 compares complete systems. To test the added value of VPM with the backbone held fixed, we compared Qwen2.5-VL-7B with and without VPM on NSD subject 1. VPM improves all nine metrics, including BLEU-4 from 22.33 to 23.71 and CIDEr from 68.91 to 72.05 (Appendix C.3, Table 10).

    • (VoRc: Weakness 2) VPM versus alternative post-training objectives.

      Our Addressing. Table 2 reports the change from supervised initialization by backbone and modality. Table 3 compares post-training objectives from the same initialized Qwen2.5-VL-3B policy on four NSD subjects. VPM improves all four reported metrics over DPO and CLIP-RL, including BLEU-4/CIDEr gains of +2.34/+4.31 over DPO and +1.39/+1.94 over CLIP-RL.

    • (VoRc: Weakness 3) Subject coverage of the system comparisons.

      Our Addressing. Table 1 uses the standard NSD subject-1 benchmark, and Table 3 averages the objective comparison over subjects 1, 2, 5, and 7. The response to VoRc also compares systems with outputs for all four subjects on the same 982 images per subject. VPM leads that comparison with 24.33 BLEU-4 and 72.26 CIDEr.

    • (Y9j4: Question 2) EEG results beyond subject 8.

      Our Addressing. We expanded the evaluation from subject 8 to all ten THINGS-EEG subjects for both backbones. All nine ten-subject means improve for each backbone. For example, CIDEr rises from 15.31 to 16.70 for SmolVLM2-2.2B and from 17.68 to 18.41 for Qwen2.5-VL-3B (Table 2 and Appendix C.2, Table 9).

AuthorsOpenReview ↗

Review and Reviewer-Author Discussion Summary (2/2)

  • Evidence for the VPM reward.

    • (Y9j4: Question 3) Dependence of the gains on stimulus matching.

      Our Addressing. We kept the generated captions fixed and reevaluated them after reassigning references across images. Over 20 reassignments, VPM’s CIDEr gain over CLIP-RL falls from +1.94 to +0.22, and its gain over DPO falls from +4.31 to +0.20. This supports a stimulus-associated component of the gains (Y9j4 response, Q3).

    • (Y9j4: Question 4) Uncertainty in the paired comparisons.

      Our Addressing. We bootstrapped the 982 test images 5,000 times, keeping the four subjects together. All six paired 95% confidence intervals for BLEU-4 and CIDEr gains over DPO, CLIP-RL, and the ODE variant lie above zero. For example, the CIDEr gain over CLIP-RL is +1.94, with a 95% interval of [0.88, 2.96] (Y9j4 response, Q4).

    • (Y9j4: Question 5) Reward sensitivity to details in similar scenes.

      Our Addressing. We swapped captions within three selected pairs that differ in surfboard activity, horse count, or bus activity, holding the target, reference, and anchor noise fixed. Matched captions score higher in both directions for all three pairs. On the image with one horse, the single-horse caption scores 1.844 versus 1.769 for the two-horse caption (Appendix C.8, Figure 8).

  • Method design and reproducibility.

    • (Y9j4: Weakness 1) Reproducibility of brain–visual alignment.

      Our Addressing. Appendix C.4 specifies the fMRI voxel vectors, the flattened 17 × 250 EEG input, and the BrainXS mapping to 16 tokens of dimension 2,048. It identifies the alignment targets for each backbone and describes how the brain tokens enter the frozen VLM during initialization.

    • (Y9j4: Weakness 2) Relation to prior brain captioning work. Our Addressing. Section 2.1 now discusses Jing et al., Horikawa, and Chen et al. Their approaches use multimodal alignment, iterative text optimization, and personalized semantic masking, respectively. VPM contributes a stimulus-anchored reward for optimizing the language policy after brain–visual alignment.

    • (Y9j4: Question 1) Choice of noisy anchors and critic cost. Our Addressing. Section 4.4 compares one, three, and five anchors and shifts the placement at fixed K = 5 (Tables 6–7). The default five-anchor window [0.1, 0.5] attains the highest BLEU-4, ROUGE-L, and CIDEr in both sweeps. The estimated critic cost is 0.278, 0.834, and 1.389 GFLOPs per caption for K = 1, 3, and 5, respectively. The critic runs only during training.

The added evaluations support improved visual faithfulness on NSD dataset and show higher mean captioning scores for both backbones on THINGS-EEG. We hope this summary of the reviews and our responses helps inform the decision. We thank the reviewers for their constructive feedback and the Area Chair, Senior Area Chair, and Program Chairs for their time and consideration.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.