- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Mind World Model (MWM), which learns from synchronized EEG and egocentric video without action labels. An EEG encoder produces a latent action that conditions prediction of future visual features. Training combines masked EEG reconstruction, EEG-image alignment, and future prediction. A Lateralization-Aware Transformer (LAT) aims to separate shared left/right activity from side-specific activity. The model is pretrained on 63.28 hours from EgoBrain and evaluated with frozen-backbone linear probes on nine classification and seven regression tasks. The paper also evaluates transfer of the frozen world model to IMU-conditioned prediction, reporting lower feature-prediction error than an inverse-dynamics latent-action baseline.
Critical strengths
-
The pretraining idea is well motivated. Using the visual consequences of everyday behavior supplies richer supervision than a small set of movement labels. The EEG window ends at the current frame, so the model does not explicitly receive the future frame through its action encoder. This offers an interesting connection between motor representation learning and world-model training.
-
The downstream evaluation is broad and reasonably controlled. All backbones receive the same task-specific EEG interval, use participant-disjoint splits, and are evaluated with frozen features over three probe seeds. MWM achieves the highest mean classification score on every listed task and the highest regression correlation on six of seven tasks. This supports the practical value of the learned EEG representations.
Critical weaknesses
The claimed symmetry properties of LAT do not follow clearly from the proposed soft-gated mirror. Equation (7) mixes unchanged and swapped features using a gate between zero and one. Applying this operation twice does not generally recover the original features: a constant gate of 0.5 averages a left/right pair rather than implementing a reversible mirror. Consequently, the invariant/equivariant interpretation of Equation (9) needs a derivation or explicit qualifications. This matters because the structural separation is a central contribution, while removing frame averaging changes average downstream performance by only 2.3% in Table 3.
Questions for authors
-
What constraints on the gate make the learned mirror reversible and justify the stated invariant/equivariant readouts? If these properties are empirical tendencies rather than guarantees, please clarify that distinction.
-
Are pretraining filters/resampling causal or applied before window extraction, and do all three IMU summaries end at the current frame? These details affect the interpretation of movement-related information and the avoidance of future-information leakage.
2 comments
Thanks for the authors' feedback. I will keep my rating but with only 1 confidence.
We thank the reviewer for the valuable feedback and would like to address the concerns below.
W1/Q1
We thank the reviewer for the careful reading. Eq. (7), written as a map on Z, indeed does not recover Z when applied twice, and we will fix the notation. The mirror is defined after the decomposition in Eq. (6): it acts on the pair (Zbi,Zlat), which is computed once and never re-split. On this pair, applying it twice recovers the input for any gate:
g(g(Zbi,Zlat))=g(Zbi, ΠZlat)=(Zbi, Π2Zlat)=(Zbi,Zlat).
In other words, the gate is applied only once, and mirroring the input simply swaps the two passes of frame averaging instead of re-splitting the features. For example, with a constant gate G=0.5, the decomposition gives (21Z,21Z), which the mirror maps to (21Z,21ΠZ) and back, rather than averaging a left/right pair. Hence, invariance and equivariance under the mirror g are guaranteed by the architecture. Fig. 4 instead examines a harder case that the architecture does not guarantee: mirroring the raw electrodes. We report both cases below, where each entry is the correlation between the features before and after the exchange (+1: unchanged; −1: sign flipped). Exchanging after the decomposition gives exactly +1 and −1, as guaranteed by the architecture.
Even under the raw exchange, the bilateral features remain almost unchanged and the lateral features largely flip sign, consistently across all subjects. This shows that the model learns the intended symmetry beyond what the architecture enforces.
We will make the definition of the mirror and the distinction between guaranteed and empirical properties explicit in the revised version.
Q2
We thank the reviewer for the question. The band-pass/notch filters and resampling are zero-phase and applied to the continuous recording before window extraction, but the resulting leakage of EEG after t is negligible and cannot reveal the target frame. The filter energy is highly concentrated around its center: only 0.4% lies more than 50 ms after t, and the remaining tail carries only sub-0.3 Hz drift. Meanwhile, the visual response to the target frame at t+0.2 appears in EEG only after an additional latency of about 50–100 ms, i.e., beyond the reach of any non-negligible filter weight. We will include the discussion in the revised version.
Using IMU measurements after the current frame is intended. In this evaluation, the IMU serves as a proxy for the real action that drives the transition from the current frame t to the target t+0.2, and this action necessarily takes place after t. Following the standard convention of action-conditioned world models that at is executed between st and st+1, the three summaries therefore cover [t−0.2,t), [t,t+0.2), and [t+0.2,t+0.4) s around this step rather than ending at t, where the two neighboring intervals hedge against the IMU–EEG/video clock misalignment of about 0.10 s. We will state the intervals explicitly in the revised version.