acceptodds
Under review as a conference paper at ICLR 2027

CortexLink: Reconstructing Long Videos from fMRI with Event-Level Semantic Consensus and Continuous Geometry

Abstract

Current fMRI-to-video systems typically reconstruct continuous visual experience as independent short clips, overlooking the event structure shared by neighboring neural measurements. This mismatch discards repeated semantic evidence and amplifies local uncertainty into content drift and geometric discontinuities across clip boundaries. We introduce CortexLink, an event-aligned framework that reorganizes clip-level predictions within stimulus shots. Its semantic branch treats decoded descriptions as noisy observations of a shared event and distills them into a consensus anchor through Minimum Bayes Risk decoding. Its geometric branch combines shot-level spatial aggregation during training with groupwise shape–pose decomposition at inference, recovering a shared subject template and a continuous pose trajectory. The resulting semantic and geometric anchors jointly guide long-video synthesis and reset at event boundaries. Experiments show consistent improvements in semantic fidelity, motion agreement, and cross-clip continuity over clip-level baselines. On shots lasting at least 10 seconds, CortexLink improves 50-way identification by \(15.5%\), while reducing Dynamic Degree error by \(51.8%\) and Boundary Seam by \(67.3%\) relative to CortexVideo. Component analyses further verify the complementary contributions of semantic consensus and continuous geometry. These results demonstrate that event structure provides an effective organizing principle for advancing fMRI reconstruction from isolated clips toward coherent long videos.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI decoding, brain-to-video reconstruction, event segmentation, minimum Bayes risk decoding, motion representation, visual neuroscience

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6ggyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CortexLink extends a clip-level fMRI decoder with shot-level caption consensus and shared-mask geometry. It combines registered shape templates with smoothed pose trajectories, reporting improved semantic identification and reduced clip-boundary discontinuities on CC2017, particularly for longer shots.

Critical strengths

The method addresses a concrete failure of independent clip reconstruction. Table 1 shows improvements in both semantic identification and continuity with the same underlying generator. The explicit shape–pose decomposition offers an interpretable way to share information while allowing objects to move across clips.

Critical weaknesses
  1. The comparison does not separate boundary information from the proposed inference method. Section 3 supplies CortexLink with a stimulus-side shot partition, whereas the principal baselines reconstruct clips independently. This disclosed conditional setting is legitimate, but the “without test-time ground truth” description needs qualification, and a simple baseline given the same boundaries is needed to establish the additional value of CortexLink.
  2. The semantic weighting is motivated by agreement rather than demonstrated reliability. Equations 2–3 favor captions close to the same candidate-set centroid used to establish consensus, so a recurring decoding error could receive high weight. Since MBR selection is established prior art, a comparison against uniform medoid selection and ordinary caption sharing would help establish whether the proposed weighting contributes beyond consensus itself.
  3. The motion metrics do not establish that the recovered trajectory is correct. Dynamic Degree discrepancy measures motion amount, while Boundary Seam measures feature discontinuity; neither tests whether motion follows the target's direction or path. Given the shared shape template and temporal regularization in Equations 5–9, a target-relative displacement measure against a simple smoothing control would distinguish accurate recovery from plausible stabilization.
  4. The ablation evidence does not establish how the two branches affect final video quality. Table 2 evaluates caption and mask intermediates, with mask IoU increasing from 0.2369 to 0.2419, rather than reporting semantic-only, geometry-only, and combined video outputs. Such a comparison under the existing generation settings would show whether both proposed components contribute to the improvements in Table 1 and whether their benefits interact.
  5. The scope of the long-video result is difficult to assess from the reported aggregation. Table 1 separates 6–8-second and at-least-10-second shots without reporting their counts or duration distributions, although the dataset contains 597 shots. The method shares conditions within shots and resets at boundaries, so these results support within-shot consistency more directly than general long-sequence reconstruction; reporting coverage would clarify the practical scope of the advance.
Questions for authors
  1. Is the intended task reconstruction with known stimulus boundaries? Under those same boundaries, how do uniform caption consensus and simple mask smoothing compare with the full method?
  2. Can you report semantic-only, geometry-only, and combined video results, including one target-relative motion measure and the number and duration distribution of shots in each evaluation bin?
5 comments

We thank the reviewer for the careful reading and for recognizing CortexLink's motivation and interpretable shape–pose decomposition. We first answer Q1 and Q2 with the requested ablations, then address W1–W5 in order.

Q1. How do uniform caption consensus and simple mask smoothing compare under the same boundaries?

We thank the reviewer for suggesting these comparisons. Yes, the intended setting is reconstruction given stimulus-derived shot boundaries. Under the same shot partition, the following caption and geometric comparisons report three-subject means.

A. Caption selection

Medium shots: 6–8 s (3–4 clips)

Caption selection2-way ↑50-way ↑EPE ↓Seam ↓CLIP-pcc ↑SSIM ↑
Random caption broadcasting0.8720.25621.5962.2000.9170.378
Uniform medoid selection0.8780.30821.4272.0630.9240.381
Centrality-weighted MBR (CortexLink)0.8800.32721.0962.0510.9270.382

Long shots: ≥10 s (≥5 clips)

Caption selection2-way ↑50-way ↑EPE ↓Seam ↓CLIP-pcc ↑SSIM ↑
Prompt broadcasting (random caption)0.8820.3208.1271.9770.9220.415
Uniform medoid selection0.9020.3788.1241.9530.9280.422
Centrality-weighted MBR (CortexLink)0.9110.3957.9881.9580.9280.423

Caption comparison. Uniform consensus improves 50-way accuracy over random broadcasting from 0.256 / 0.320 to 0.308 / 0.378 on medium/long shots. Centrality-weighted MBR further raises it to 0.327 / 0.395, absolute gains of 1.9%/1.7% over Uniform, with lower EPE and higher SSIM in both bins. This is consistent with our motivation in the Introduction, where we argue that observations in an event's interior tend to share stable semantics, whereas those near boundaries can mix neighboring events and introduce semantic drift. Weighting captions by agreement with the candidate semantic centroid emphasizes recurring event content and reduces the influence of divergent descriptions, providing a rationale for the additional gains over uniform consensus.

B. Geometric components, including simple smoothing

Relation to the geometric formulation (Eqs. 4–9). All rows retain MaskBank (Eq. 4) and start from the resulting decoder's mask predictions Mt\mathbf M_t. Full Trace registers them to a shared template Ts\mathbf T_s using transforms GtG_t (Eqs. 5–7), then fits a pose trajectory and renders the geometric condition:

\sum_{t=1}^{L_s}\bar q_t\\|\boldsymbol\pi_t-\mathbf o_t\\|_2^2 +\lambda\sum_{t=2}^{L_s-1}\\|\boldsymbol\pi_{t-1}-2\boldsymbol\pi_t+\boldsymbol\pi_{t+1}\\|_2^2.$$ In Eq. (8), $\boldsymbol\pi_t$ is the fitted pose, $\mathbf o_t=-\log G_t$ the observed pose, $\bar q_t$ its reliability weight, and $L_s$ the frame count. The two terms fit the observations and smooth the trajectory, balanced by $\lambda$. $$\widetilde{\mathbf M}_t= \alpha\mathcal W(\mathbf T_s,\exp(\boldsymbol\pi_t)) +(1-\alpha)\mathcal W(\mathbf M_t,\exp(\boldsymbol\pi_t)G_t).$$ In Eq. (9), $\mathcal W$ warps a mask: the first term moves the shared template along the trajectory, and the second retains frame-specific shape information; $\alpha$ balances them. | Geometric conditioning | What is removed or replaced | | --- | --- | | Decoded masks (no Trace) | Use $\widetilde{\mathbf M}_t=\mathbf M_t$ directly, bypassing Trace in Eqs. (5)–(9); retain MaskBank. | | Shot-averaged mask | Replace Trace with the unregistered mean $\widetilde{\mathbf M}_t=L_s^{-1}\sum_u\mathbf M_u$, shared across frames; remove time-varying geometry. | | Mask smoothing (EMA) | Apply EMA directly to the decoded masks; replace registration, the shared template, and pose-trajectory rendering with pixel-wise temporal smoothing. | | Trace without trajectory smoothing | Set $\boldsymbol\pi_t=\mathbf o_t$, equivalent to $\lambda=0$ in Eq. (8); retain registration, the template, and both terms of Eq. (9). | | Fixed-shape Trace | Set $\alpha=1$ in Eq. (9), removing the frame-specific shape term; retain the smoothed motion trajectory. | | Full Trace (CortexLink) | Retain registration, trajectory smoothing, and both rendering terms. | **Medium shots: 6–8 s (3–4 clips)** | Geometric conditioning | 2-way ↑ | 50-way ↑ | ΔDD ↓ | EPE ↓ | CLIP-pcc ↑ | SSIM ↑ | | ------------------------------------- | --------- | --------- | --------- | ---------- | ---------- | --------- | | Decoded masks (no Trace) | 0.868 | 0.287 | 0.071 | 21.398 | 0.923 | 0.379 | | Shot-averaged mask | 0.876 | 0.300 | 0.061 | 21.355 | 0.922 | 0.376 | | Mask smoothing (EMA) | 0.872 | 0.312 | 0.058 | 21.181 | 0.922 | 0.378 | | Trace without trajectory smoothing | 0.875 | 0.299 | 0.074 | 21.464 | 0.923 | 0.378 | | Fixed-shape Trace | 0.877 | 0.283 | **0.054** | 21.240 | 0.922 | 0.380 | | **Full Trace (CortexLink)** | **0.880** | **0.327** | 0.065 | **21.096** | **0.927** | **0.382** | **Long shots: ≥10 s (≥5 clips)** | Geometric conditioning | 2-way ↑ | 50-way ↑ | ΔDD ↓ | EPE ↓ | CLIP-pcc ↑ | SSIM ↑ | | ------------------------------------- | --------- | --------- | --------- | --------- | ---------- | --------- | | Decoded masks (no Trace) | 0.890 | 0.351 | 0.048 | 8.248 | 0.925 | 0.419 | | Shot-averaged mask | 0.894 | 0.366 | **0.038** | 8.150 | 0.926 | **0.424** | | Mask smoothing (EMA) | 0.901 | 0.367 | 0.053 | 8.298 | 0.925 | 0.423 | | Trace without trajectory smoothing | 0.887 | 0.356 | 0.061 | 8.104 | 0.926 | 0.423 | | Fixed-shape Trace | 0.897 | 0.365 | 0.050 | 8.106 | 0.926 | 0.423 | | **Full Trace (CortexLink)** | **0.911** | **0.395** | 0.044 | **7.988** | **0.928** | 0.423 | **Geometry comparison.** Shot averaging and mask smoothing both use information across frames within an event, yet provide smaller identification gains than full Trace. Relative to mask smoothing, CortexLink raises 50-way accuracy from **0.312 / 0.367 to 0.327 / 0.395** and reduces EPE from **21.181 / 8.298 to 21.096 / 7.988** on medium/long shots. By combining shape–pose decomposition and a shared template with a continuous pose trajectory, full Trace achieves the highest 2-way and 50-way accuracy and the lowest EPE among these geometric controls in both bins.

Q2. What are the branch ablation results and the duration distribution?

Thank you for this helpful suggestion.

A. Final-video branch ablation

We compare CortexVideo, geometry only, semantics only, and full CortexLink on Subject 1. EPE provides the target-relative motion measure by comparing optical flow estimated from reconstructed and target videos.

Medium shots: 6–8 s (3–4 clips)

Configuration2-way ↑50-way ↑ΔDD ↓EPE ↓
CortexVideo0.8780.2650.11021.841
+ Geometry only0.8690.2930.09721.162
+ Semantic only0.8850.3010.04622.562
CortexLink0.8920.3550.09120.677

Long shots: ≥10 s (≥5 clips)

Configuration2-way ↑50-way ↑ΔDD ↓EPE ↓
CortexVideo0.8790.3020.0728.391
+ Geometry only0.8740.2960.0648.109
+ Semantic only0.9000.3420.0979.819
CortexLink0.9050.3860.0347.165

Branch comparison. Relative to CortexVideo, the semantic branch achieves absolute 50-way accuracy gains of 3.6%/4.0% on medium/long shots, while the geometric branch reduces EPE from 21.841 / 8.391 to 21.162 / 8.109. Combining them yields the highest 50-way accuracy (0.355 / 0.386) and lowest EPE (20.677 / 7.165) in both bins, and leads all four metrics on long shots. These complementary trends support combining semantic consensus and geometric modeling to improve both content recovery and motion agreement in the final videos.

B. Shot counts and duration distribution in the evaluated bins

The two duration bins evaluated in Table 1 contain 389 clips across 81 shots, totaling 778 seconds of stimuli. Each shot spans 3–13 consecutive 2-second decoding clips, with durations ranging from 6 to 26 seconds. All three subjects share this stimulus partition.

Evaluation binShotsClipsTotal durationShot duration distribution (count)
Medium — 6–8 s46154308 s6 s (30); 8 s (16)
Long — ≥10 s35235470 s10 s (10); 12 s (10); 14 s (5); 16 s (7); 20/22/26 s (1 each)

Contribution of the full method. Our motivation is that visual events span multiple fixed decoding clips, whose shared content and evolving geometry can be recovered more effectively by combining evidence across observations. CortexLink organizes this evidence into a consensus semantic anchor, a shared shape template, and a continuous pose trajectory. The comparisons above show that this event-level design improves semantic identification and target-relative motion agreement over the requested simple alternatives, while the branch ablations support combining both anchors to further improve final-video reconstruction.

Reviewer_n1xaOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CortexLink, an event-aligned framework for fMRI-to-video reconstruction that organizes clip-level predictions within stimulus shots rather than decoding them independently. Its semantic branch treats decoded clip descriptions as noisy observations of a shared event and distills a consensus anchor through weighted Minimum Bayes Risk decoding over TF–IDF utilities. Its geometric branch combines a training-time event-aligned MaskBank with groupwise shape–pose decomposition at inference, recovering a shared subject template and a continuous pose trajectory that resets at shot boundaries. The two anchors jointly condition a video generator built on CortexVideo. On CC2017 across three subjects (597 shots), CortexLink improves 50-way identification on 10s+ shots from 0.322 to 0.372 (+15.5%), while cutting Dynamic Degree error by 51.8% and Boundary Seam by 67.3%. Ablations show BLEU-4 rising from 0.0476 to 0.0586 and CIDEr-D from 0.7738 to 0.9713, with mask IoU improving and Mask Seam dropping 38.3%. Limitations include shots only approximating cognitive events, a detail ceiling inherited from decoded masks, attenuation of high-frequency motion by constant-velocity smoothing, and generalization evidenced by only three subjects.

Critical strengths

++I have gone through this paper carefully. Overall, the motivation and core idea of this work are quite good, and the subsequent experiments are also valid and well-supported.

Critical weaknesses

The following are the questions and contents that require the authors to provide strict responses and make corresponding revisions:

--The writing and organization of this paper are of very poor quality. The distinctions among clip, event and shot are not clearly defined, and many variables are introduced abruptly without any prior explanation. Additionally, why does a mask suddenly appear in the upper branch of Figure 2? The motivation stated in the introduction section is extremely ambiguous, and the rationale for the proposed technique is unclear, presented abruptly without any logical transition. It is recommended that the authors carefully and rigorously polish the entire manuscript.

--As shown in Figure 2, the text descriptions of multiple clips select the maximum possibility through similarity calculation. Won't this process discard a large amount of detailed semantic information? Besides, the calculation method of weight w marked in Formula 3 and the corresponding part in the figure do not match: one uses an equal sign while the other uses a proportional sign, but the actual calculation logic is the same. The authors need to treat this work with more rigor.

Questions for authors

See Weakness.

In conclusion, if the authors can satisfactorily address my concerns, I will recommend ‌strong acceptance‌, as this topic is of great significance!

4 comments

We thank the reviewer for the careful reading and the constructive suggestions on presentation and methodological clarity. To address these concerns, we have carefully revised the manuscript and provide detailed clarifications below. We hope these results and revisions could help address the reviewer’s concerns.

W1. Writing organization, definitions of clip / shot / event, variable introductions

We thank the reviewer for these suggestions on presentation. We have revised the Introduction, Method, and Figure 2 to clarify the terminology and key notation.

1.Definitions of clip, event, and shot. We now clarify these terms in the motivation in Section 1 (Introduction) and restate their definitions in the overview of Section 3:

  • Clip (i∈{1,…,N}i \in \{1,\dots,N\}): An operational input unit defined by preprocessing: a fixed 2-second sampling window (F=6F=6 frames) paired with one fMRI measurement xi\mathbf x_i.
  • Visual Event: A conceptual unit denoting a temporally contiguous viewing interval with coherent scene semantics, principal subject identity, and motion dynamics.
  • Shot (s=[as,bs)s = [a_s, b_s)): A contiguous interval between adjacent editing cuts in the stimulus video. Under our partition, each shot comprises several consecutive clips, with Ks=bs−asK_s=b_s-a_s clips and Ls=KsFL_s=K_sF frames. Because visual-event boundaries are not directly annotated, we use these clearly delimited shots as practical proxies for visual events.

We retain event-level to describe the research motivation and modeling framework, and use shot to specify the actual stimulus partition, the scope of computation, and the evaluation units. Thus, an event is the unit of coherent content that we model; a shot is the interval between editing cuts used for implementation and evaluation.

2.Clarity and consistency of presentation. We have clarified the notation in Section 3 by defining symbols at first use or directly below the relevant equations, and revised the Figure 2. We will also standardize the mathematically equivalent weight expressions in Equation (3) and Figure 2 to the explicit normalized form.

3.Motivation and logical flow of the Introduction. We revised Section 1 to clarify that a fixed 2-second clip is an input unit imposed by preprocessing, but it is not necessarily the appropriate inference unit for continuous visual experience. Within a visual event, neighboring fMRI measurements provide complementary noisy evidence about shared semantics and a continuously evolving subject state; near event boundaries, hemodynamic mixing further increases clip-level uncertainty. Decoding each clip independently therefore wastes within-event evidence and can turn local uncertainty into cross-clip semantic drift and spatial jumps. CortexLink addresses this mismatch by using shots as practical proxies for events: semantic evidence and geometric states are integrated within each shot and reset at its boundary. This rationale directly motivates our two complementary components, event-level semantic consensus and continuous geometric modeling.

W2. Does MBR consensus discard detailed semantic information?

1.Recovering shared semantics from repeated noisy observations. Thank you for raising this question. Clip descriptions decoded from fMRI contain both valid content and prediction errors, and the correctness of an individual detail is unknown at inference. CortexLink therefore uses cross-clip agreement as evidence: a detail recurring across several decoded descriptions contributes to their semantic similarity and supports candidates containing it. Equation (2) selects a complete caption by maximizing its weighted agreement with the other candidates. Repeatedly supported details thus have a greater opportunity to enter the shared caption, while conflicting predictions receive less support.

2.A semantic anchor with complementary visual conditions. The shared caption is intended to stabilize the principal subject, scene, and action context. Visual keyframe and the frame-wise blurry video stream provide complementary appearance information, while time-varying Trace masks guide subject location and pose. Figure 2 illustrates this division using the Subject 3 turtle example. The decoded candidates include “a dog playing with a toy on the floor” and “a school of fish swimming underwater”; the selected caption, “a turtle swimming in the ocean,” retains the stimulus's subject (turtle), action (swimming), and scene context (ocean), correcting the conflicting dog/fish predictions. This shared semantic anchor works alongside the visual conditions that supply additional frame-dependent information.

3.Quantitative evidence from caption recovery and video reconstruction. Averaged across all three subjects on long shots, weighted MBR improves all four reported caption metrics over the original decoded captions. At the video level, full CortexLink improves three-subject mean 50-way semantic identification over CortexVideo from 28.40% to 32.75% on 6–8 s shots and from 32.20% to 39.48% on ≥10 s shots. The caption-level gains support the recovery of discriminative semantic content through consensus, while the video-level results show the reconstruction benefit of the complete CortexLink framework. Together, these results support the practical benefit of correcting decoding errors while retaining repeatedly supported semantic content.

Caption decoding

MetricOriginal decoded captionsWeighted MBR captions
BLEU-1 ↑0.26430.2810
BLEU-4 ↑0.04280.0597
ROUGE-L ↑0.25550.2801
CIDEr-D ↑0.92671.2566

Video reconstruction

Shot durationMetricCortexVideoCortexLink
6–8 s50-way (%) ↑28.4032.75
≥10 s50-way (%) ↑32.2039.48
Reviewer_n1xaOpenReview ↗

res.

Thank you very much for the authors' response. The concerns I raised have been fully addressed, and I would like to further raise my score accordingly.

Thank you for constructive feedback. We are glad that our response and revisions have addressed your concerns. Your constructive suggestions have helped us clarify the motivation and improve the presentation of our work, and we sincerely appreciate the time and care you devoted to reviewing our paper.

Reviewer_pizDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the reconstruction of semantically consistent and temporally continuous long videos from fMRI, and proposes CortexLink. Building on clip-level predictions from CortexVideo, the method aggregates captions within visual shots and uses weighted Minimum Bayes Risk decoding to select event-level semantic anchors. It also combines a training-time MaskBank with groupwise shape–pose decomposition and trajectory smoothing at inference to construct geometric conditions. Experiments on CC2017 report improvements in semantic identification, motion agreement, and cross-clip continuity.

Critical strengths
  1. Clear problem motivation. The paper identifies the mismatch between fixed-length decoding clips and natural visual events as a source of semantic drift and geometric discontinuity. Organizing repeated neural observations at the event level is a well-motivated response to this problem.

  2. Coherent method design. The semantic branch uses MBR to select a consensus description from captions within the same shot. The geometric branch separates shared subject shape from time-varying pose, using MaskBank and a continuous trajectory to improve spatial stability. The two branches jointly guide video synthesis.

  3. Multi-level experimental evaluation. The paper reports semantic, pixel-level, and temporal metrics, duration-stratified comparisons, caption- and mask-level component analyses, and qualitative examples and failure cases across subjects.

Critical weaknesses

Major Weaknesses:

  1. Ground-truth stimulus shot boundaries are used at test time, raising concerns about applicability and comparison fairness. Appendix B.1 derives shot boundaries from HSV differences between frames of the original viewed stimulus and uses this partition for semantic aggregation and trajectory resets at inference. In practical fMRI-to-video reconstruction, the shot boundaries of the unknown target video are generally unavailable; the baselines also do not receive this additional input. It is therefore unclear how much of the improvement comes from event-level modeling versus the stimulus-derived boundary prior. The authors should clarify the information available at test time and evaluate boundaries predicted from fMRI or model outputs, no boundary information, and perturbed boundaries, reporting the effects on the main metrics.

  2. The main components lack isolated end-to-end ablations. Table 2 mainly compares intermediate caption and trace-mask results before and after processing. It does not separately remove semantic consensus, MaskBank, groupwise registration, or trajectory smoothing under a common final-video evaluation, nor does it adequately isolate interactions between shared text prompts and geometric conditions. Additional single-component and combination ablations, together with simple shot-level averaging or prompt broadcasting, fixed-shape, and smoothing-only controls under the same generation budget, are needed to establish the contribution of each component.

  3. Improved cross-clip continuity does not necessarily imply more accurate recovery of real motion. The second-order regularizer in Eq. (8) suppresses acceleration changes, while Eq. (9) blends a shared template with frame-specific masks. These operations may reduce Boundary Seam while also smoothing away genuine non-rigid motion. Mask IoU in Table 2 rises only from 0.2369 to 0.2419, providing limited evidence of improved geometric accuracy. The authors should evaluate optical-flow or subject-trajectory agreement with the ground-truth video, turning and fast-motion cases, identity consistency, and sensitivity to λ and α, to distinguish smoother motion from more accurate motion.

Minor Weaknesses:

  1. Unresolved cross-references in the appendix. Appendix A.3 refers to MaskBank using “Sec. ??”, and Appendix B.2 uses “Sec. ??” when discussing MBR (p. 16). These cross-references should be corrected, and the manuscript should be checked for similar issues.

  2. The description of “long-video generation” should be more precise. Appendix B.4 states that synthesis remains clip-based, with semantic and geometric conditions shared across clips within a shot. The contribution is therefore better characterized as cross-clip conditioning and coordination rather than generating a complete shot jointly in one pass. This distinction should be explicit in the method and conclusion.

Questions for authors
  1. Test-time shot boundaries. Can the authors clarify why boundaries derived from the original stimulus video are permissible at inference? Do the main gains remain when boundaries must be predicted from fMRI or generated content, are unavailable, or contain detection errors?

  2. End-to-end contribution of each component. Can the authors report final-video results when semantic consensus, MaskBank, groupwise registration, and trajectory smoothing are individually removed, as well as comparisons with simpler shared-prompt or smoothing-only alternatives under matched generation settings?

  3. Motion fidelity versus smoothing. Can the authors assess subject trajectories or optical flow against the ground-truth stimulus, particularly for fast or non-rigid motion, and report the sensitivity of motion fidelity and continuity to λ and α?

5 comments

We thank the reviewer for recognizing CortexLink's motivation and dual-branch design. We clarify the boundary input and report final-video branch, caption, and geometric comparisons, including optical-flow agreement with the target videos below.

W1 & Q1. Boundary input and event-level modeling

We thank the reviewer for raising the question of boundary information. Previous fMRI-to-video studies typically divide video sequences into fixed-duration clips and reconstruct them independently. For a long shot depicting a coherent visual event, however, different clips provide complementary observations of shared scene content, subject identity, and evolving motion. Independent decoding leaves these relationships unmodeled, allowing uncertainty in individual predictions to produce semantic drift and spatial discontinuities. CortexLink therefore expands the modeling unit to the complete video segment corresponding to a coherent event and integrates the associated neural observations through semantic consensus and continuous geometric modeling.

CC2017 provides temporally ordered fMRI signals elicited while subjects watch natural-video sequences, making it possible to study this relationship across clips. Given the stimulus shot boundaries, we organize the video and its aligned fMRI observations into corresponding sequences and study how jointly using the neural sequence improves reconstruction of the complete video segment. Our primary task is event-level video reconstruction under known boundaries. Events define the modeling objective, and shots provide the operational partition. The boundary input specifies which observations are grouped together; ground-truth captions, masks, and motion trajectories are not used at inference.

To address the reviewer's concern about unavailable target-video boundaries, we explore extending CortexLink by predicting boundaries from fMRI and using the predicted segments to guide reconstruction. On Subject 1, we trained a lightweight binary boundary head whose predictions determine the groups for caption aggregation. The two qualitative examples in Appendix F, Figure 6 support the feasibility of connecting this prediction head to CortexLink's semantic aggregation and video synthesis stages, providing a concrete step toward reconstruction without supplied video boundaries.

We also compare against simple alternatives given the same stimulus-side shot boundaries. Prompt broadcasting achieves 0.256 / 0.320 in 50-way accuracy on medium/long shots, and shot averaging achieves 0.300 / 0.366, compared with 0.327 / 0.395 for CortexLink. These comparisons support the benefit of the proposed evidence integration under the given boundaries. With boundary information held constant, the additional gains support the contribution of CortexLink's semantic consensus and geometric modeling beyond the boundary prior itself.

W2 & Q2. Component contributions

We appreciate the suggestion to isolate the contributions of the individual components.

A. Semantic and geometric branches

Medium shots — 6–8 s (3–4 clips)

Configuration2-way ↑50-way ↑ΔDD ↓EPE ↓
CortexVideo0.8780.2650.11021.841
Geometry only0.8690.2930.09721.162
Semantic only0.8850.3010.04622.562
CortexLink0.8920.3550.09120.677

Long shots — ≥10 s (≥5 clips)

Configuration2-way ↑50-way ↑ΔDD ↓EPE ↓
CortexVideo0.8790.3020.0728.391
Geometry only0.8740.2960.0648.109
Semantic only0.9000.3420.0979.819
CortexLink0.9050.3860.0347.165

Branch comparison. Relative to CortexVideo, the semantic branch achieves absolute 50-way accuracy gains of 3.6%/4.0% on medium/long shots, while the geometric branch reduces EPE from 21.841 / 8.391 to 21.162 / 8.109. Combining them yields the highest 50-way accuracy (0.355 / 0.386) and lowest EPE (20.677 / 7.165) in both bins, and leads all four metrics on long shots. These complementary trends support combining semantic consensus and geometric modeling to improve both content recovery and motion agreement in the final videos.

B. Caption selection — three-subject means

Medium shots — 6–8 s (3–4 clips)

Caption selection2-way ↑50-way ↑EPE ↓Seam ↓CLIP-pcc ↑SSIM ↑
Prompt broadcasting (random caption)0.8720.25621.5962.2000.9170.378
Uniform medoid selection0.8780.30821.4272.0630.9240.381
Centrality-weighted MBR (CortexLink)0.8800.32721.0962.0510.9270.382

Long shots — ≥10 s (≥5 clips)

Caption selection2-way ↑50-way ↑EPE ↓Seam ↓CLIP-pcc ↑SSIM ↑
Prompt broadcasting (random caption)0.8820.3208.1271.9770.9220.415
Uniform medoid selection0.9020.3788.1241.9530.9280.422
Centrality-weighted MBR (CortexLink)0.9110.3957.9881.9580.9280.423

Caption comparison. Uniform consensus improves 50-way accuracy over random broadcasting from 0.256 / 0.320 to 0.308 / 0.378 on medium/long shots. Centrality-weighted MBR further raises it to 0.327 / 0.395, absolute gains of 1.9%/1.7% over Uniform, with lower EPE and higher SSIM in both bins. This is consistent with our motivation in the Introduction, where we argue that observations in an event's interior tend to share stable semantics, whereas those near boundaries can mix neighboring events and introduce semantic drift. Weighting captions by agreement with the candidate semantic centroid emphasizes recurring event content and reduces the influence of divergent descriptions, providing a rationale for the additional gains over uniform consensus.

C. Geometric components: MaskBank and Trace

MaskBank contribution. We toggle the inference prior of the same trained decoder and report three-subject means for medium (6–8 s) and long (≥10 s) shots.

MaskBankTraceMedium-shot Mask IoU ↑Long-shot Mask IoU ↑
OffOff0.26130.2191
OnOff0.27650.2411
OffOn0.27110.2469
OnOn0.28110.2552

The final-video comparison retains full Trace.

ShotsConfiguration2-way ↑50-way ↑EPE ↓Seam ↓
MediumWithout MaskBank prior0.8720.31621.7142.090
MediumFull CortexLink0.8800.32721.0962.051
LongWithout MaskBank prior0.9010.3708.1561.975
LongFull CortexLink0.9110.3957.9881.958

MaskBank improves mask IoU with and without Trace. With full Trace, it raises 50-way accuracy by 1.10 / 2.44 percentage points on medium/long shots and lowers EPE in both bins, supporting its contribution to final-video reconstruction.

Relation to the Trace formulation (Eqs. 5–9). The final-video Trace comparisons below retain MaskBank (Eq. 4) and start from the resulting decoder's mask predictions Mt\mathbf M_t. Full Trace registers them to a shared template Ts\mathbf T_s using transforms GtG_t (Eqs. 5–7), then fits a pose trajectory and renders the geometric condition.

\sum_{t=1}^{L_s}\bar q_t\\|\boldsymbol\pi_t-\mathbf o_t\\|_2^2 +\lambda\sum_{t=2}^{L_s-1}\\|\boldsymbol\pi_{t-1}-2\boldsymbol\pi_t+\boldsymbol\pi_{t+1}\\|_2^2.$$ In Eq. (8), $\boldsymbol\pi_t$ is the fitted pose, $\mathbf o_t=-\log G_t$ the observed pose, $\bar q_t$ its reliability weight, and $L_s$ the frame count. The two terms fit the observations and smooth the trajectory, balanced by $\lambda$. $$\widetilde{\mathbf M}_t= \alpha\mathcal W(\mathbf T_s,\exp(\boldsymbol\pi_t)) +(1-\alpha)\mathcal W(\mathbf M_t,\exp(\boldsymbol\pi_t)G_t).$$ In Eq. (9), $\mathcal W$ warps a mask. The first term moves the shared template along the trajectory, and the second retains frame-specific shape information; $\alpha$ balances them. | Geometric conditioning | What is removed or replaced | | --- | --- | | Decoded masks (no Trace) | Use $\widetilde{\mathbf M}_t=\mathbf M_t$ directly, bypassing Trace in Eqs. (5)–(9); retain MaskBank. | | Shot-averaged mask | Replace Trace with the unregistered mean $\widetilde{\mathbf M}_t=L_s^{-1}\sum_u\mathbf M_u$, shared across frames; remove time-varying geometry. | | Mask smoothing (EMA) | Apply EMA directly to the decoded masks; replace registration, the shared template, and pose-trajectory rendering with pixel-wise temporal smoothing. | | Trace without trajectory smoothing | Set $\boldsymbol\pi_t=\mathbf o_t$, equivalent to $\lambda=0$ in Eq. (8); retain registration, the template, and both terms of Eq. (9). | | Fixed-shape Trace | Set $\alpha=1$ in Eq. (9), removing the frame-specific shape term; retain the smoothed motion trajectory. | | Full Trace (CortexLink) | Retain registration, trajectory smoothing, and both rendering terms. | **Medium shots — 6–8 s (3–4 clips)** | Geometric conditioning | 2-way ↑ | 50-way ↑ | ΔDD ↓ | EPE ↓ | CLIP-pcc ↑ | SSIM ↑ | |:---|---:|---:|---:|---:|---:|---:| | Decoded masks (no Trace) | 0.868 | 0.287 | 0.071 | 21.398 | 0.923 | 0.379 | | Shot-averaged mask | 0.876 | 0.300 | 0.061 | 21.355 | 0.922 | 0.376 | | Mask smoothing (EMA) | 0.872 | 0.312 | 0.058 | 21.181 | 0.922 | 0.378 | | Trace without trajectory smoothing | 0.875 | 0.299 | 0.074 | 21.464 | 0.923 | 0.378 | | Fixed-shape Trace | 0.877 | 0.283 | **0.054** | 21.240 | 0.922 | 0.380 | | **Full Trace (CortexLink)** | **0.880** | **0.327** | 0.065 | **21.096** | **0.927** | **0.382** | **Long shots — ≥10 s (≥5 clips)** | Geometric conditioning | 2-way ↑ | 50-way ↑ | ΔDD ↓ | EPE ↓ | CLIP-pcc ↑ | SSIM ↑ | |:---|---:|---:|---:|---:|---:|---:| | Decoded masks (no Trace) | 0.890 | 0.351 | 0.048 | 8.248 | 0.925 | 0.419 | | Shot-averaged mask | 0.894 | 0.366 | **0.038** | 8.150 | 0.926 | **0.424** | | Mask smoothing (EMA) | 0.901 | 0.367 | 0.053 | 8.298 | 0.925 | 0.423 | | Trace without trajectory smoothing | 0.887 | 0.356 | 0.061 | 8.104 | 0.926 | 0.423 | | Fixed-shape Trace | 0.897 | 0.365 | 0.050 | 8.106 | 0.926 | 0.423 | | **Full Trace (CortexLink)** | **0.911** | **0.395** | 0.044 | **7.988** | **0.928** | 0.423 | **Geometry comparison.** Shot averaging and mask smoothing both use information across frames within an event, yet provide smaller identification gains than full Trace. Relative to mask smoothing, CortexLink raises 50-way accuracy from **0.312 / 0.367 to 0.327 / 0.395** and reduces EPE from **21.181 / 8.298 to 21.096 / 7.988** on medium/long shots. By combining shape–pose decomposition and a shared template with a continuous pose trajectory, full Trace achieves the highest 2-way and 50-way accuracy and the lowest EPE among these geometric controls in both bins.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.