- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CortexLink extends a clip-level fMRI decoder with shot-level caption consensus and shared-mask geometry. It combines registered shape templates with smoothed pose trajectories, reporting improved semantic identification and reduced clip-boundary discontinuities on CC2017, particularly for longer shots.
Critical strengths
The method addresses a concrete failure of independent clip reconstruction. Table 1 shows improvements in both semantic identification and continuity with the same underlying generator. The explicit shape–pose decomposition offers an interpretable way to share information while allowing objects to move across clips.
Critical weaknesses
- The comparison does not separate boundary information from the proposed inference method. Section 3 supplies CortexLink with a stimulus-side shot partition, whereas the principal baselines reconstruct clips independently. This disclosed conditional setting is legitimate, but the “without test-time ground truth” description needs qualification, and a simple baseline given the same boundaries is needed to establish the additional value of CortexLink.
- The semantic weighting is motivated by agreement rather than demonstrated reliability. Equations 2–3 favor captions close to the same candidate-set centroid used to establish consensus, so a recurring decoding error could receive high weight. Since MBR selection is established prior art, a comparison against uniform medoid selection and ordinary caption sharing would help establish whether the proposed weighting contributes beyond consensus itself.
- The motion metrics do not establish that the recovered trajectory is correct. Dynamic Degree discrepancy measures motion amount, while Boundary Seam measures feature discontinuity; neither tests whether motion follows the target's direction or path. Given the shared shape template and temporal regularization in Equations 5–9, a target-relative displacement measure against a simple smoothing control would distinguish accurate recovery from plausible stabilization.
- The ablation evidence does not establish how the two branches affect final video quality. Table 2 evaluates caption and mask intermediates, with mask IoU increasing from 0.2369 to 0.2419, rather than reporting semantic-only, geometry-only, and combined video outputs. Such a comparison under the existing generation settings would show whether both proposed components contribute to the improvements in Table 1 and whether their benefits interact.
- The scope of the long-video result is difficult to assess from the reported aggregation. Table 1 separates 6–8-second and at-least-10-second shots without reporting their counts or duration distributions, although the dataset contains 597 shots. The method shares conditions within shots and resets at boundaries, so these results support within-shot consistency more directly than general long-sequence reconstruction; reporting coverage would clarify the practical scope of the advance.
Questions for authors
- Is the intended task reconstruction with known stimulus boundaries? Under those same boundaries, how do uniform caption consensus and simple mask smoothing compare with the full method?
- Can you report semantic-only, geometry-only, and combined video results, including one target-relative motion measure and the number and duration distribution of shots in each evaluation bin?
5 comments
B. Geometric components, including simple smoothing
Relation to the geometric formulation (Eqs. 4–9). All rows retain MaskBank (Eq. 4) and start from the resulting decoder's mask predictions . Full Trace registers them to a shared template using transforms (Eqs. 5–7), then fits a pose trajectory and renders the geometric condition:
\sum_{t=1}^{L_s}\bar q_t\\|\boldsymbol\pi_t-\mathbf o_t\\|_2^2 +\lambda\sum_{t=2}^{L_s-1}\\|\boldsymbol\pi_{t-1}-2\boldsymbol\pi_t+\boldsymbol\pi_{t+1}\\|_2^2.$$ In Eq. (8), $\boldsymbol\pi_t$ is the fitted pose, $\mathbf o_t=-\log G_t$ the observed pose, $\bar q_t$ its reliability weight, and $L_s$ the frame count. The two terms fit the observations and smooth the trajectory, balanced by $\lambda$. $$\widetilde{\mathbf M}_t= \alpha\mathcal W(\mathbf T_s,\exp(\boldsymbol\pi_t)) +(1-\alpha)\mathcal W(\mathbf M_t,\exp(\boldsymbol\pi_t)G_t).$$ In Eq. (9), $\mathcal W$ warps a mask: the first term moves the shared template along the trajectory, and the second retains frame-specific shape information; $\alpha$ balances them. | Geometric conditioning | What is removed or replaced | | --- | --- | | Decoded masks (no Trace) | Use $\widetilde{\mathbf M}_t=\mathbf M_t$ directly, bypassing Trace in Eqs. (5)–(9); retain MaskBank. | | Shot-averaged mask | Replace Trace with the unregistered mean $\widetilde{\mathbf M}_t=L_s^{-1}\sum_u\mathbf M_u$, shared across frames; remove time-varying geometry. | | Mask smoothing (EMA) | Apply EMA directly to the decoded masks; replace registration, the shared template, and pose-trajectory rendering with pixel-wise temporal smoothing. | | Trace without trajectory smoothing | Set $\boldsymbol\pi_t=\mathbf o_t$, equivalent to $\lambda=0$ in Eq. (8); retain registration, the template, and both terms of Eq. (9). | | Fixed-shape Trace | Set $\alpha=1$ in Eq. (9), removing the frame-specific shape term; retain the smoothed motion trajectory. | | Full Trace (CortexLink) | Retain registration, trajectory smoothing, and both rendering terms. | **Medium shots: 6–8 s (3–4 clips)** | Geometric conditioning | 2-way ↑ | 50-way ↑ | ΔDD ↓ | EPE ↓ | CLIP-pcc ↑ | SSIM ↑ | | ------------------------------------- | --------- | --------- | --------- | ---------- | ---------- | --------- | | Decoded masks (no Trace) | 0.868 | 0.287 | 0.071 | 21.398 | 0.923 | 0.379 | | Shot-averaged mask | 0.876 | 0.300 | 0.061 | 21.355 | 0.922 | 0.376 | | Mask smoothing (EMA) | 0.872 | 0.312 | 0.058 | 21.181 | 0.922 | 0.378 | | Trace without trajectory smoothing | 0.875 | 0.299 | 0.074 | 21.464 | 0.923 | 0.378 | | Fixed-shape Trace | 0.877 | 0.283 | **0.054** | 21.240 | 0.922 | 0.380 | | **Full Trace (CortexLink)** | **0.880** | **0.327** | 0.065 | **21.096** | **0.927** | **0.382** | **Long shots: ≥10 s (≥5 clips)** | Geometric conditioning | 2-way ↑ | 50-way ↑ | ΔDD ↓ | EPE ↓ | CLIP-pcc ↑ | SSIM ↑ | | ------------------------------------- | --------- | --------- | --------- | --------- | ---------- | --------- | | Decoded masks (no Trace) | 0.890 | 0.351 | 0.048 | 8.248 | 0.925 | 0.419 | | Shot-averaged mask | 0.894 | 0.366 | **0.038** | 8.150 | 0.926 | **0.424** | | Mask smoothing (EMA) | 0.901 | 0.367 | 0.053 | 8.298 | 0.925 | 0.423 | | Trace without trajectory smoothing | 0.887 | 0.356 | 0.061 | 8.104 | 0.926 | 0.423 | | Fixed-shape Trace | 0.897 | 0.365 | 0.050 | 8.106 | 0.926 | 0.423 | | **Full Trace (CortexLink)** | **0.911** | **0.395** | 0.044 | **7.988** | **0.928** | 0.423 | **Geometry comparison.** Shot averaging and mask smoothing both use information across frames within an event, yet provide smaller identification gains than full Trace. Relative to mask smoothing, CortexLink raises 50-way accuracy from **0.312 / 0.367 to 0.327 / 0.395** and reduces EPE from **21.181 / 8.298 to 21.096 / 7.988** on medium/long shots. By combining shape–pose decomposition and a shared template with a continuous pose trajectory, full Trace achieves the highest 2-way and 50-way accuracy and the lowest EPE among these geometric controls in both bins.Q2. What are the branch ablation results and the duration distribution?
Thank you for this helpful suggestion.
A. Final-video branch ablation
We compare CortexVideo, geometry only, semantics only, and full CortexLink on Subject 1. EPE provides the target-relative motion measure by comparing optical flow estimated from reconstructed and target videos.
Medium shots: 6–8 s (3–4 clips)
| Configuration | 2-way ↑ | 50-way ↑ | ΔDD ↓ | EPE ↓ |
|---|---|---|---|---|
| CortexVideo | 0.878 | 0.265 | 0.110 | 21.841 |
| + Geometry only | 0.869 | 0.293 | 0.097 | 21.162 |
| + Semantic only | 0.885 | 0.301 | 0.046 | 22.562 |
| CortexLink | 0.892 | 0.355 | 0.091 | 20.677 |
Long shots: ≥10 s (≥5 clips)
| Configuration | 2-way ↑ | 50-way ↑ | ΔDD ↓ | EPE ↓ |
|---|---|---|---|---|
| CortexVideo | 0.879 | 0.302 | 0.072 | 8.391 |
| + Geometry only | 0.874 | 0.296 | 0.064 | 8.109 |
| + Semantic only | 0.900 | 0.342 | 0.097 | 9.819 |
| CortexLink | 0.905 | 0.386 | 0.034 | 7.165 |
Branch comparison. Relative to CortexVideo, the semantic branch achieves absolute 50-way accuracy gains of 3.6%/4.0% on medium/long shots, while the geometric branch reduces EPE from 21.841 / 8.391 to 21.162 / 8.109. Combining them yields the highest 50-way accuracy (0.355 / 0.386) and lowest EPE (20.677 / 7.165) in both bins, and leads all four metrics on long shots. These complementary trends support combining semantic consensus and geometric modeling to improve both content recovery and motion agreement in the final videos.
B. Shot counts and duration distribution in the evaluated bins
The two duration bins evaluated in Table 1 contain 389 clips across 81 shots, totaling 778 seconds of stimuli. Each shot spans 3–13 consecutive 2-second decoding clips, with durations ranging from 6 to 26 seconds. All three subjects share this stimulus partition.
| Evaluation bin | Shots | Clips | Total duration | Shot duration distribution (count) |
|---|---|---|---|---|
| Medium — 6–8 s | 46 | 154 | 308 s | 6 s (30); 8 s (16) |
| Long — ≥10 s | 35 | 235 | 470 s | 10 s (10); 12 s (10); 14 s (5); 16 s (7); 20/22/26 s (1 each) |
Contribution of the full method. Our motivation is that visual events span multiple fixed decoding clips, whose shared content and evolving geometry can be recovered more effectively by combining evidence across observations. CortexLink organizes this evidence into a consensus semantic anchor, a shared shape template, and a continuous pose trajectory. The comparisons above show that this event-level design improves semantic identification and target-relative motion agreement over the requested simple alternatives, while the branch ablations support combining both anchors to further improve final-video reconstruction.
We thank the reviewer for the careful reading and for recognizing CortexLink's motivation and interpretable shape–pose decomposition. We first answer Q1 and Q2 with the requested ablations, then address W1–W5 in order.
Q1. How do uniform caption consensus and simple mask smoothing compare under the same boundaries?
We thank the reviewer for suggesting these comparisons. Yes, the intended setting is reconstruction given stimulus-derived shot boundaries. Under the same shot partition, the following caption and geometric comparisons report three-subject means.
A. Caption selection
Medium shots: 6–8 s (3–4 clips)
Long shots: ≥10 s (≥5 clips)
Caption comparison. Uniform consensus improves 50-way accuracy over random broadcasting from 0.256 / 0.320 to 0.308 / 0.378 on medium/long shots. Centrality-weighted MBR further raises it to 0.327 / 0.395, absolute gains of 1.9%/1.7% over Uniform, with lower EPE and higher SSIM in both bins. This is consistent with our motivation in the Introduction, where we argue that observations in an event's interior tend to share stable semantics, whereas those near boundaries can mix neighboring events and introduce semantic drift. Weighting captions by agreement with the candidate semantic centroid emphasizes recurring event content and reduces the influence of divergent descriptions, providing a rationale for the additional gains over uniform consensus.