acceptodds
Withdrawn from ICLR 2027

CALIBRA-fMRI: Text for Signals Without Captions, Read by a Frozen Video-Language Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal language models answer many kinds of question in one output space, and extending them to scientific signals requires text that rarely exists and must be generated. Generated text can often be answered without the non-text input, a shortcut usually found only after a model has been trained on it. Task fMRI comes with a controlled experiment and rich metadata, from which answers can be computed and the shortcut controlled at construction. We build CALIBRA-fMRI, whose items span the experimental design, the signal, behavior and the participant in a design space defined by a rule-based item generator. Each item carries a signal-free accuracy, fixed before training by a rule that answers it without the recording, and a model trained on the text alone stays at this level (.252 against .259). Without paired fMRI–text data at pretraining scale, we first verify that the image encoder of a frozen video-language model, never trained on fMRI, is competitive with fMRI foundation models under a linear probe. Building on it, we develop an fMRI aggregator that pools each four-dimensional volume through a cortical atlas, compressing a 15-minute recording from millions of patch tokens into thousands. Coupled with the frozen language model, it forms RIVET, which takes fMRI inputs of any length while training only 21.2M parameters. Two frozen fMRI foundation models with trained heads each answer one side, relations within a segment or participant identity, whereas RIVET answers both when trained on each and, trained on all levels at once, exceeds the signal-free accuracy on 9 of 14 items. It orders states in time without per-step labels and, from three segments on, learns to compare segments given separately, a comparison it carries to unseen paradigms. Trained on sex instead, it retains much less of the moment-to-moment state, consistent with how the two foundation models split as released, so what a model can answer depends on what its input keeps and what its items ask. The construction needs only a record that fixes the answers, offering multimodal language models a way to build corpora that require their non-text input by design.

Primary area:
applications to neuroscience & cognitive science
Keywords:
vision-language model, multimodal alignment, benchmarks and evaluation, fMRI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vHqTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes CALIBRA-fMRI, a question–answer corpus generated by rules from HCP-YA task fMRI. They also propose a framework named RIVET, which feeds fMRI to a frozen Qwen3-VL-2B and train on the data.

Critical strengths

The method seems efficient, RIVET is trained only 21.2M parameters. Pooling 2,592 patch tokens per frame into 7 fits whole 900-frame runs on one 40 GB GPU, and one generative objective covers every answer form that the baselines need separate heads and hand-written rules for.

Critical weaknesses
  1. I can barely understand the writing. After reading the abstract for several times I still don't understand what problem the paper tries to solve and what the proposed method is. Sentences like "Generated text can often be answered without the non-text input, a shortcut usually found only after a model has been trained on it." just don't make sense. The prose is dense, nonstandard terms such as cells, hosts and recoverability pile up, and many key results appear only in a 20-page appendix. I would advise the authors to improve their writing if this is not an autoresearch paper.

  2. Figures like Figure 1 and 3 are of low quality. The texts are cramped together, and it's difficult to digest what the author trying to say

  3. The novelty is very limited. fMRI-LM (though the author cited it) already generates template text from the computed imaging quantities. The change here is just extending to task fMRI and a per-item f. The paper also overstates the gap since fMRI-LM already describes temporal properties. Additionally, using a pretrained vision encoder is not new, as in [1][2]

[1] Kim et al., “Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?” [2] Han et al., “OneLLM: One Framework to Align All Modalities with Language”

  1. "f" is explained poorly. It's just a standard baseline (accuracy of always giving the most common training answer in the test set) that has a new name.

  2. RIVET does not beat a simple foundation model baseline. The frozen BrainLM with a small head is better.

  3. The scope is narrow, limited to HCP-YA.

Questions for authors

The paper is fundamentally flawed so I don't have any clarification question

Reviewer_FwkWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CALIBRA-fMRI derives question-answer items from task-fMRI records and measures normalized gain over a training-fitted signal-free reference ff. RIVET combines frozen vision/LM components with a 21.2M-parameter aggregator. It performs well on state ordering and separately trained identity comparison, with weaker results elsewhere. The contributions are an auditable construction and a unified interface claiming parity, not universal superiority.

Critical strengths

S1. Labels, eligibility checks and the pre-evaluation reference are explicit, with failed items disclosed. This makes corpus construction and predictable-answer shortcuts auditable.

S2. State-sequence exact match is 0.667 versus 0.325 for the frame-count control (Table 22). Collapsed match is 0.806 versus 0.830 for the per-step-supervised BrainLM GRU. The odd-one-out task also has a useful one-host chance control.

Critical weaknesses

W1. Shortcut coverage remains incomplete. The paper defines ff as a training-fitted empirical reference, not a proven upper bound on blind prediction. The frame-count rule exceeds ff on state ordering, which is appropriately controlled, but equivalent shape-aware evidence is missing for the headline working-memory order item. The text-only control fixes frames to zero and count to 16 while retaining the frozen LM. This tests recording removal for RIVET, but excludes shape information and stronger question/context learners. The concern is the scope of the shortcut interpretation, not that ff is incorrectly defined.

W2. The vision tower's added value is not isolated. A matched parcel-series connector into the same frozen LM is missing. Baseline input windows and checkpoint-selection rules also differ, and Appendix J.2 attributes much of the identity contrast to retained run means. These differences limit architectural attribution. BrainLM leads on seven of eight Table 3 forms, but lack of universal superiority is not a rejection reason given the paper's explicit parity claim. Intended identity/trait information should also be distinguished from evidence of functional-activity decoding.

Questions for authors
  1. Shortcuts: Do the main gains survive a question/context learner and a recording-free baseline retaining true frame/segment counts, especially for working-memory order?
  2. Interface: How does a matched parcel-series connector perform under the same selection rule? Alternatively, narrow the claim to a demonstrated workable interface.
  3. Consistency: Please reconcile +0.329 versus +0.269 for seen triples and clarify that success at three hosts means four total segments, not three.
Reviewer_JAptOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CALIBRA-fMRI, a benchmark consisting of approximately 2.23 million QA items, utilizing the experimental design, signal statistics, behavior, and individual characteristics of HCP-YA task fMRI from 1,097 participants. Additionally, RIVET is an interface connecting fMRI to frozen Qwen3-VL-2B, compressing the features of each volume into 7 network tokens based on a brain atlas. It processes the entire fMRI time series by training only an aggregator with 21.2 million parameters.

Critical strengths

[1] The authors processed over 900 frames of recordings using a trainable aggregator and extracted features from fMRI volumes via VIT without pre-training a separate fMRI-specific encoder.

[2] The authors constructed their own large-scale question-answering dataset with signal-free accuracy.

[3] The authors conducted extensive experiments and validations to ensure the model actually relies on brain signal information to generate answers.

Critical weaknesses

[1] The authors control for shortcuts using signal-free accuracy (f) and text-only controls, but there is unclear how much segment-specific network information contributes to L1 relational performance. These controls do not distinguish between directly reading segment-specific network relationships and identifying the task condition before predicting condition-typical patterns.

[2] [minor] The main QA evaluations are conducted on HCP-YA, while ABIDE and ADHD-200 are used for linear-probe evaluations of encoder representations. Evaluating the model on an external task-fMRI dataset with compatible questions would strengthen the evidence for generalization across acquisition settings and experimental designs.

[3] [minor] The added value of the vision encoder over direct parcel inputs remains unclear. While the encoder probes and aggregator ablations are informative, the paper does not compare RIVET with raw and temporally standardized parcel time series mapped to the same frozen LLM through a capacity-matched projector. This baseline would help establish the benefit of the pretrained vision encoder beyond the aggregator and language model.

Questions for authors

[1] Regarding Weakness 1, could the authors provide a baseline with condition only to assess how much of the L1 relational performance can be explained by condition specific network patterns rather than segment?

[2] Regarding Weakness 2, do the authors plan to evaluate the generalization of the fMRI-QA pipeline to an external task-fMRI dataset?

[3] Regarding Weakness 3, could the authors show the contribution of the pretrained vision encoder by comparing RIVET with direct parcel-level inputs mapped to the same frozen LLM through a parameter-matched projector?

[4] As mentioned in the Discussion, including a matched-budget LoRA comparison and evaluating with family splits could further strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.