acceptodds
Under review as a conference paper at ICLR 2027

Mind World Models: Joint Learning of Motor EEG Representations and Action-Conditioned World Dynamics

Abstract

Motor decoding from electroencephalography (EEG) supports non-invasive brain–computer interfaces, yet existing methods either rely on costly cue-based trials and coarse categorical labels or use pretraining objectives that do not directly capture how movement unfolds, limiting their ability to learn transferable motor representations. To this end, we propose the Mind World Model (MWM), a framework that learns motor EEG representations from scalable naturalistic EEG and egocentric video without action annotations. MWM treats EEG observed up to the current frame as the latent action of a world model: a mind model encodes the EEG, and a world model uses it to predict the representation of a future frame, yielding a movement-grounded pretraining objective. We further introduce the Lateralization-Aware Transformer (LAT), which uses frame averaging to separate shared bilateral from side-specific lateral features, encoding hemispheric correspondence to reduce the burden of learning left–right relationships from data. MWM pretraining produces two models: a mind model for downstream motor decoding and a world model that generalizes from EEG-derived actions to real-world action signals, which also offers a scalable and flexible path to collect egocentric data for embodied AIs. Across 9 classification and 7 regression tasks, the mind model improves task-averaged balanced accuracy by 5.9% and Pearson correlation by 16.2% relative to the strongest single baseline in each task family. Compared with an inverse-dynamics latent-action baseline, the world model reduces L1 error by 23.2% within recordings and 11.0% on held-out subjects.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, world model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MZzgOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Mind World Model (MWM), which learns from synchronized EEG and egocentric video without action labels. An EEG encoder produces a latent action that conditions prediction of future visual features. Training combines masked EEG reconstruction, EEG-image alignment, and future prediction. A Lateralization-Aware Transformer (LAT) aims to separate shared left/right activity from side-specific activity. The model is pretrained on 63.28 hours from EgoBrain and evaluated with frozen-backbone linear probes on nine classification and seven regression tasks. The paper also evaluates transfer of the frozen world model to IMU-conditioned prediction, reporting lower feature-prediction error than an inverse-dynamics latent-action baseline.

Critical strengths
  1. The pretraining idea is well motivated. Using the visual consequences of everyday behavior supplies richer supervision than a small set of movement labels. The EEG window ends at the current frame, so the model does not explicitly receive the future frame through its action encoder. This offers an interesting connection between motor representation learning and world-model training.

  2. The downstream evaluation is broad and reasonably controlled. All backbones receive the same task-specific EEG interval, use participant-disjoint splits, and are evaluated with frozen features over three probe seeds. MWM achieves the highest mean classification score on every listed task and the highest regression correlation on six of seven tasks. This supports the practical value of the learned EEG representations.

Critical weaknesses

The claimed symmetry properties of LAT do not follow clearly from the proposed soft-gated mirror. Equation (7) mixes unchanged and swapped features using a gate between zero and one. Applying this operation twice does not generally recover the original features: a constant gate of 0.5 averages a left/right pair rather than implementing a reversible mirror. Consequently, the invariant/equivariant interpretation of Equation (9) needs a derivation or explicit qualifications. This matters because the structural separation is a central contribution, while removing frame averaging changes average downstream performance by only 2.3% in Table 3.

Questions for authors
  1. What constraints on the gate make the learned mirror reversible and justify the stated invariant/equivariant readouts? If these properties are empirical tendencies rather than guarantees, please clarify that distinction.

  2. Are pretraining filters/resampling causal or applied before window extraction, and do all three IMU summaries end at the current frame? These details affect the interpretation of movement-related information and the avoidance of future-information leakage.

2 comments

We thank the reviewer for the valuable feedback and would like to address the concerns below.

W1/Q1

The soft gate in Eq. (7) makes the mirror non-reversible (e.g., a gate of 0.5 averages a left/right pair), which matters since removing frame averaging costs only 2.3% in Table 3. Are the invariant/equivariant properties guaranteed or empirical?

We thank the reviewer for the careful reading. Eq. (7), written as a map on Z\mathbf{Z}, indeed does not recover Z\mathbf{Z} when applied twice, and we will fix the notation. The mirror is defined after the decomposition in Eq. (6): it acts on the pair (Zbi,Zlat)(\mathbf{Z}^{\mathrm{bi}},\mathbf{Z}^{\mathrm{lat}}), which is computed once and never re-split. On this pair, applying it twice recovers the input for any gate:

g(g(Zbi,Zlat))=g(Zbi, ΠZlat)=(Zbi, Π2Zlat)=(Zbi,Zlat).g\big(g(\mathbf{Z}^{\mathrm{bi}},\mathbf{Z}^{\mathrm{lat}})\big) = g(\mathbf{Z}^{\mathrm{bi}},\ \Pi\mathbf{Z}^{\mathrm{lat}}) = (\mathbf{Z}^{\mathrm{bi}},\ \Pi^2\mathbf{Z}^{\mathrm{lat}}) = (\mathbf{Z}^{\mathrm{bi}},\mathbf{Z}^{\mathrm{lat}}).

In other words, the gate is applied only once, and mirroring the input simply swaps the two passes of frame averaging instead of re-splitting the features. For example, with a constant gate G=0.5\mathbf{G}=0.5, the decomposition gives (12Z,12Z)(\tfrac12\mathbf{Z},\tfrac12\mathbf{Z}), which the mirror maps to (12Z,12ΠZ)(\tfrac12\mathbf{Z},\tfrac12\Pi\mathbf{Z}) and back, rather than averaging a left/right pair. Hence, invariance and equivariance under the mirror gg are guaranteed by the architecture. Fig. 4 instead examines a harder case that the architecture does not guarantee: mirroring the raw electrodes. We report both cases below, where each entry is the correlation between the features before and after the exchange (+1: unchanged; −1: sign flipped). Exchanging after the decomposition gives exactly +1 and −1, as guaranteed by the architecture.

ExchangeDatasetBilateralLateral
After decompositionEgoBrain+1.00−1.00
After decompositionPhysioNet-MI+1.00−1.00
Raw electrodesEgoBrain+0.98−0.76
Raw electrodesPhysioNet-MI+0.98−0.70

Even under the raw exchange, the bilateral features remain almost unchanged and the lateral features largely flip sign, consistently across all subjects. This shows that the model learns the intended symmetry beyond what the architecture enforces.

We will make the definition of the mirror and the distinction between guaranteed and empirical properties explicit in the revised version.

Q2

Are pretraining filters/resampling causal or applied before window extraction?

We thank the reviewer for the question. The band-pass/notch filters and resampling are zero-phase and applied to the continuous recording before window extraction, but the resulting leakage of EEG after tt is negligible and cannot reveal the target frame. The filter energy is highly concentrated around its center: only 0.4% lies more than 50 ms after tt, and the remaining tail carries only sub-0.3 Hz drift. Meanwhile, the visual response to the target frame at t+0.2t+0.2 appears in EEG only after an additional latency of about 50–100 ms, i.e., beyond the reach of any non-negligible filter weight. We will include the discussion in the revised version.

Do all three IMU summaries end at the current frame?

Using IMU measurements after the current frame is intended. In this evaluation, the IMU serves as a proxy for the real action that drives the transition from the current frame tt to the target t+0.2t+0.2, and this action necessarily takes place after tt. Following the standard convention of action-conditioned world models that ata_t is executed between sts_t and st+1s_{t+1}, the three summaries therefore cover [t−0.2,t)[t-0.2,t), [t,t+0.2)[t,t+0.2), and [t+0.2,t+0.4)[t+0.2,t+0.4) s around this step rather than ending at tt, where the two neighboring intervals hedge against the IMU–EEG/video clock misalignment of about 0.10 s. We will state the intervals explicitly in the revised version.

Thanks for the authors' feedback. I will keep my rating but with only 1 confidence.

Reviewer_1phPOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses joint representation learning between EEG signals and egocentric videos. It proposes a self-supervised method that uses paired EEG signals and egocentric videos to predict future frames from the current frame, conditioned on the EEG signal up to the current time step. The method is scalable and does not require manual annotations. The paper also proposes a network architecture called the Lateralization-Aware Transformer (LAT), which uses prior knowledge of EEG signals related to motor lateralization. The experimental results show that the proposed method achieves strong performance on various downstream tasks and benchmarks.

Critical strengths
  • The paper is well written and easy to follow. In particular, the Introduction clearly explains the motivation for joint representation learning between EEG signals and egocentric videos.
  • The proposed approach is reasonable and has several advantages. It is scalable because it does not require manual annotations, it is not limited to a closed-set vocabulary, and it avoids information leakage during training.
  • LAT is an interesting way to take into account the prior knowledge of EEG signals related to motor lateralization. The analysis and visualization in Sec. 4.4 show that LAT behaves as expected.
Critical weaknesses
  • The ablation study in Sec. 4.5 suggests that the benefit of the proposed frame prediction is limited. Removing the alignment loss decreases performance by 17.6%, while removing the prediction loss decreases it by only 2.6%. The idea itself is interesting, but the strong performance of MWM may come mainly from the standard masked EEG patch reconstruction loss (L_mask) and the alignment loss (L_align) rather than from the proposed prediction loss. The contribution of this paper would be more solid if the benefit of the prediction loss were shown more clearly.
  • Related to the above point, one way to show this would be to present qualitative results that show in which cases the proposed method works well and in which cases it fails. Currently, the paper reports only quantitative results and gives no concrete examples. A discussion based on qualitative results would make the paper more convincing.
  • The proposed method is trained to predict the frame 200 ms ahead, and this time span appears to be fixed. How was this value chosen? Is it supported by neuroscientific findings, or was it chosen empirically? Intuitively, the appropriate time span may vary across tasks, environments, participants, and so on. How does performance change when the time span is changed?
  • Why is AdaLM used instead of more common conditioning mechanisms such as cross-attention? Is this choice supported by neuroscientific findings, or was it validated empirically through ablation studies?
Questions for authors

See weaknesses.

4 comments

We thank the reviewer for the valuable feedback and would like to address the concerns below.

W1

The benefit of the prediction loss appears limited: removing alignment costs 17.6% while removing prediction costs only 2.6%, so the performance may come mainly from masked reconstruction and alignment.

We thank the reviewer for the comment. We agree that alignment contributes the most to downstream transfer, but prediction plays an essential and complementary role:

  • The average hides task-specific effects. Removing prediction costs 5–11% on four tasks: WAYGALReg (−10.6%), Jeong2020 (−5.7%), Schwarz (−5.2%), and ForenzoFutVec (−4.7%). On the other eight tasks the change is at most 3.6% in magnitude (PhysioNet-MI −1.0%, Kaya5F −1.1%, FineMI −1.0%, Ofner2017MI −3.0%, SrisReachME −0.4%, OfnerApertureME −2.1%, BiomechMag +0.6%, ForceGameReg +3.6%), so the small average mainly reflects these tasks. Notably, Schwarz decodes the grasp type before movement onset and ForenzoFutVec predicts future velocity, where anticipating upcoming movement is helpful.
  • Prediction is what makes MWM a world model. Without it, there is no action-conditioned predictor to transfer. As shown in our updated Table 4, MWM's action gap is 2–3× that of an identically trained action-free predictor.
  • Prediction is required for the lateralization structure. In our analysis, the LAT gate opens (mean 0.85) only when both prediction and alignment are used; without prediction it collapses to 0.04, and the lateral/bilateral separation disappears.

We will add this discussion in the revised version.

W2

The paper reports only quantitative results; qualitative examples of where the method works and fails would be more convincing.

We thank the reviewer for the suggestion. Comparing MWM with the strongest baseline on each task of our main results reveals a clear pattern of where our training framework works and where it is less effective:

  • Works best: motor execution, especially continuous kinematics. The largest gains appear on executed movements, e.g., +52% on OfnerApertureME, +46% on ForceGameReg, and +18% on OfnerKinME relative to the strongest baseline, with an average of +21% over the execution regression tasks. This matches our design, as MWM learns from the visual consequences of self-executed everyday actions.
  • Smaller gains: motor imagery and fine-grained classification. MWM still ranks first on every motor-imagery task, but the average gain drops to +4%, and to about +1% on fine-grained tasks such as five-finger imagery (Kaya5F) and FineMI. Imagined movements produce no visual consequence, and finger-level differences are barely visible in egocentric video, so our pretraining signal covers them less.
  • Failure case: WAYGALReg is the only task where MWM falls behind the strongest baseline (0.433 vs. 0.460 for REVE).

We will add this discussion to the revised version.

W3

How was the 200 ms prediction horizon chosen, and how does performance change with other horizons?

We thank the reviewer for the question. The 200 ms horizon was fixed a priori rather than tuned on downstream tasks. It equals the temporal resolution of our model: each EEG token spans 0.2 s (40 samples at 200 Hz, chosen to keep a 5 Hz spectral resolution within a token), and the frame grid is aligned to it, so 200 ms is the shortest horizon the model can resolve. Moreover, among the horizons the model can resolve, the visual change at 200 ms is the most tied to the wearer's own hand motion. On EgoBrain (8 subjects), we correlate the DINOv2 change between the current and future frames with the WiLoR hand speed over the same interval:

HorizonLateral: L–R location of change vs. R–L hand speedMagnitude: amount of change vs. total hand speed
0.2 s0.580.84
0.4 s0.530.83
0.6 s0.470.82
1.0 s0.420.81

Both correlations decrease monotonically with the horizon, and 0.2 s gives the highest lateral correlation in all 8 subjects, suggesting that longer horizons increasingly mix in visual changes unrelated to the wearer's own action.

We will add this analysis in the revised version.

W4

Why is AdaLN used instead of more common conditioning such as cross-attention?

We thank the reviewer for the question. In MWM, the EEG plays the role of the action in an action-conditioned world model, and AdaLN is a standard way to inject actions in such models, e.g., LeWorldModel (Maes et al., 2026), following the conditioning design of DiT (Peebles & Xie, 2023). We chose it for two reasons:

  • It makes the EEG act as an action rather than as extra content. With AdaLN, the EEG can only modulate how the current frame evolves through a compact vector, rather than being retrieved token by token as with cross-attention. This keeps the EEG in the slot a recorded action would occupy, and lets us compare with the latent-action model under an identical action interface (Table 4).
  • It is stable and lightweight. With zero initialization, each block starts as the identity, so the predictor initially copies the current frame and the EEG only needs to supply the change. It also adds no attention over EEG tokens.

We will clarify this choice in the revised version.

Thank you for the detailed response.

I now understand that the prediction loss is essential for training the world model, and the updated Table 4 shows that MWM has a larger action gap than the action-free predictor. I am convinced by the world-model contribution.

W1/W2

My concern in W1 and W2 was whether, where, and why the prediction loss contributes to the mind model's EEG representations, i.e., to downstream motor decoding. The prediction loss need not help every task, but then a coherent explanation of which tasks it helps and why is required. The current explanation does not provide one.

  1. The explanation that the prediction loss helps tasks requiring anticipation of future movement is inconsistent with other results. According to Sec. 4.2, SrisReachME also predicts future reaching speed, yet removing prediction changes its performance by only −0.4%. The drop on Schwarz, cited as an example (0.575±0.035 → 0.545±0.027), is comparable to the standard deviations over three seeds. Moreover, WAYGALReg, where removing prediction causes the largest drop, is left unexplained. Sec. 4.2 hypothesizes that visual prediction provides a less direct supervisory signal for the force variations in this task, so it is not obvious why the prediction loss matters most here.

  2. The explanation in the response to W2 does not account for the effect of the prediction loss. The response attributes MWM's largest gains to motor execution, especially continuous kinematics (OfnerApertureME +52%, ForceGameReg +46%). However, in Table 3, removing prediction barely changes performance on these tasks (OfnerApertureME −2.1%) and even improves ForceGameReg (+3.6%), whereas removing alignment causes large drops (−76% and −35%). These gains therefore come mainly from alignment, not from prediction.

  3. In W2, I asked for concrete samples showing where the method works and where it fails, but the response provides only task-level aggregates, which do not show in which movements or situations the prediction loss helps.

In short, the prediction loss has a small average effect (−2.6%) concentrated on a few tasks, and the rebuttal does not provide a consistent account of this pattern. The claim that MWM learns motor EEG representations by predicting the visual consequences of behavior is therefore not sufficiently supported.

W3

I understand the rationale for 200 ms. However, the correlation analysis concerns the relation between the prediction target (changes in visual features) and hand motion; it does not show how the horizon affects the EEG representations or downstream performance. Since the response to W1 attributes part of the benefit of the prediction loss to anticipating future movement, the horizon should be a key factor, and this remains unverified.

W4

I understand the rationale for AdaLN, in particular that it allows all models in Table 4 to be compared under the same action interface.

Overall

My initial rating of weak accept reflected the interesting idea behind the approach and its well-motivated design, together with the expectation that the unclear results would be clarified in the rebuttal. The world-model contribution has been strengthened by Table 4. However, for one of the main contributions, learning motor EEG representations by predicting the visual consequences of behavior, the rebuttal does not provide a coherent explanation, as discussed above, and the claim is not sufficiently supported. I am therefore currently inclined to lower my rating to weak reject. I would appreciate hearing how the authors view the scope of the claims about the contribution of the prediction loss in the paper, which I will take into account in my final rating.

We thank the reviewer for the follow-up and are glad that Table 4 strengthens the world-model contribution.

W1/W2

The anticipation explanation is inconsistent: SrisReachME also predicts future speed but barely changes without prediction, the Schwarz drop is within seed variation, and WAYGALReg, where prediction matters most, is left unexplained.

We thank the reviewer for this careful analysis, which sharpened our explanation. Alignment provides the foundation on nearly all tasks, and prediction adds on top of it where the decoded label concerns what the action does to the world.

  • Alignment matches the EEG with the current egocentric frame, which shows the scene and what the body is doing in it (hands, posture, and the activity the scene implies): a rich label of the body’s motor state according to the scene.
  • Prediction must explain how the next frame differs. The effect of an action on the world is visible only as such a change: the same frame (a hand on a cup) precedes lifting, lowering, or holding the cup, and only the EEG can tell them apart.

Table 3 bears this out (labels classified from dataset descriptions alone):

LabelTaskw/o predictionw/o alignment
WorldWAYGALReg (forces on a lifted object)−10.6%−28.2%
WorldJeong2020 (object-defined grasp classes)−5.7%−6.3%
WorldSchwarz (jar vs. spoon, pre-movement)−5.2%−4.5%
WorldForenzoFutVec (velocity of the steered cursor)−4.7%−15.0%
Body8 tasks (reach, aperture, wrist, thumb force; finger, joint, limb classes)−3.0% to +3.6%+5.5% to −76.0%

The four world-referenced tasks show the four largest drops without prediction, while all eight body-referenced tasks change by at most 3.6%. It is worth noting that the two effects are unrelated across tasks (Spearman ρ = −0.20): the tasks that depend most on alignment (OfnerApertureME, ForceGameReg, SrisReachME) are barely affected by prediction; prediction contributes something alignment does not provide. This explains SrisReachME (the speed of the participant's own finger) and WAYGALReg.

The largest gains of MWM (OfnerApertureME, ForceGameReg) come from alignment rather than prediction, as removing prediction barely changes them.

We agree that alignment is highly effective, as Table 3 reports, and explained why above. Prediction adds on top of it on world-referenced targets.

Concrete samples showing in which movements or situations the prediction loss helps are still missing.

In the Jeong2020 test confusion matrices (3 seeds, 300 trials per class), the gain lies in the object-defined grasp classes, not in the body-defined ones:

Jeong2020 classesMWMw/o prediction
Grasp trials recognized as a grasp (object-defined)0.4320.363
Reach and twist classes (body-defined), recall0.1170.120

Grasp recognition improves in all three seeds, and the Jeong2020 gain is reproduced by every checkpoint trained with both objectives, including three horizon variants below. Conversely, prediction does not help where the label is a reading of the participant's own body, even when that reading is shown on screen: wrist speed in BiomechMag (0.157 vs. 0.158) and thumb force in ForceGameReg (0.165 vs. 0.171). We will add this analysis to the appendix.

W3

The correlation analysis concerns the prediction target, not how the horizon affects the EEG representations or downstream performance.

We pretrained MWM with horizons of 0.4s and 0.6s under the identical recipe and evaluated downstream tasks (linear probing, 3 seeds; 0.2 s is the model in the paper; all 16 tasks will be added to the appendix):

Task0.2 s0.4 s0.6 s
OfnerApertureME0.2000.2290.159
OfnerKinME0.2040.1890.199
BiomechMag0.1570.1650.159
Kaya5F0.3380.3390.347
Ofner2017ME0.2020.2090.199
Ofner2017MI0.1650.1580.162
Jeong20200.1580.1580.160

Each horizon has its strengths, and none is uniformly best. Our 0.2 s step lies within the range used by recent action-conditioned world models, which predict at steps of 0.2–0.25 s (4–5 fps), including V-JEPA 2-AC (Assran et al., 2025), Navigation World Models (Bar et al., CVPR 2025), PEVA (Bai et al., 2025), and Cosmos on Bridge (NVIDIA, 2025).

Overall, regarding the scope of our claims, we will make the respective roles explicit in the revised version: alignment provides the foundation and most of the downstream gains, while prediction adds on top of it where the label is defined by what the action does to the outside world, and trains the world model that transfers to real-action conditioning (Table 4).

Reviewer_7rTLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mind World Models (MWM), a framework that jointly learns motor EEG representations and visual dynamics from synchronized EEG and egocentric video without explicit action labels. An EEG encoder maps observations up to the current frame into a latent action that conditions prediction of frozen DINOv2 features 0.2 seconds into the future. Training combines masked EEG reconstruction, cross-modal alignment, and future-feature prediction. The paper also introduces the Lateralization-Aware Transformer (LAT), which aims to separate bilateral and lateral features through learned gating and paired mirror transformations.

MWM is pretrained on 63.28 hours of EgoBrain recordings from 40 participants. Evaluation includes frozen-backbone linear probing on nine classification and seven regression tasks with participant-disjoint splits, component ablations, lateralization analyses, and transfer of the pretrained world model to IMU-conditioned prediction of future visual features. MWM achieves improved task-averaged motor-decoding performance relative to the evaluated EEG foundation-model baselines and lower prediction error than the evaluated world-model baselines.

Critical strengths
  1. An interesting annotation-free pretraining formulation. MWM uses EEG observed up to the current frame to condition future visual-feature prediction, connecting EEG representation learning with naturalistic behavioral observations without explicit action labels. This is a promising alternative to label-dependent motor pretraining and is the paper's strongest conceptual contribution.

  2. Broad evaluation across motor-decoding tasks. Frozen-backbone evaluation covers nine classification and seven regression tasks using participant-disjoint splits, with improvements in task-averaged performance over the evaluated EEG foundation models. The results suggest that the learned representations are useful across a range of motor-decoding tasks, rather than only on a single benchmark.

Critical weaknesses
  1. LAT's claimed symmetry properties are not established by its formulation. In Eqs. (6)–(9), the learned soft-gated mirror operation does not generally satisfy P2=I\mathcal P^2=I, so the subsequent averaging does not automatically guarantee the stated invariance and equivariance. Moreover, Figure 4 tests raw-channel exchange, which Appendix D distinguishes from the learned training transformation. This leaves a gap between the architectural construction and the symmetry properties claimed for it.

  2. The evidence for transferable action-conditioned dynamics remains limited. Table 3 shows that removing prediction has a modest and task-dependent effect on downstream performance. In Table 4, shuffling IMU inputs increases MWM's L1 prediction error by only about 1–2% relative to the matched-input error. The evaluation also lacks a current-state copying baseline and a comparably trained action-free predictor, making it difficult to determine whether the reported gains depend on action-conditioned dynamics or can largely be explained by prediction from the current visual state. This limits the evidence for the paper's central interpretation of predictive pretraining.

Questions for authors
  1. Clarification — LAT's symmetry guarantees: Does the implementation impose constraints on the learned gate (G) that ensure P2=I\mathcal P^2=I ? More generally, what transformation laws are intended for the bilateral and lateral output blocks, and under what conditions do Eqs. (6)–(9) satisfy them? Please clarify which properties are guaranteed by the architecture and which are supported only empirically.
  2. Targeted validation — action-dependent prediction: Does MWM retain its advantage over a current-state feature-copying baseline and an action-free predictor trained with comparable capacity and budget? Please also report uncertainty estimates for the matched-versus-shuffled IMU error gap in Table 4, ideally accounting for variation across participants. These comparisons would help determine whether the reported gains genuinely reflect action-conditioned dynamics rather than primarily exploiting information in the current visual state.
  3. Clarification — temporal availability of IMU inputs: What are the exact time intervals of the three consecutive IMU summaries described in Appendix C, relative to the current frame at (t) and the target frame at (t+0.2)? Do any summaries include measurements acquired after (t)? Please clarify the temporal alignment of all IMU inputs to establish what information is available to the predictor at inference time.
6 comments

We thank the reviewer for the valuable feedback and would like to address the concerns below.

W1/Q1

LAT's soft-gated mirror in Eqs. (6)–(9) does not generally satisfy P2=I\mathcal{P}^2=I, so the averaging does not automatically guarantee the stated invariance and equivariance. Does the implementation constrain G\mathbf{G}, and which properties are guaranteed by the architecture versus supported only empirically?

We thank the reviewer for the careful reading. Eq. (7), written as a map on Z\mathbf{Z}, indeed does not recover Z\mathbf{Z} when applied twice, and we will fix the notation. The mirror is defined after the decomposition in Eq. (6): it acts on the pair (Zbi,Zlat)(\mathbf{Z}^{\mathrm{bi}},\mathbf{Z}^{\mathrm{lat}}), which is computed once and never re-split. On this pair, applying it twice recovers the input for any gate:

g(g(Zbi,Zlat))=g(Zbi, ΠZlat)=(Zbi, Π2Zlat)=(Zbi,Zlat).g\big(g(\mathbf{Z}^{\mathrm{bi}},\mathbf{Z}^{\mathrm{lat}})\big) = g(\mathbf{Z}^{\mathrm{bi}},\ \Pi\mathbf{Z}^{\mathrm{lat}}) = (\mathbf{Z}^{\mathrm{bi}},\ \Pi^2\mathbf{Z}^{\mathrm{lat}}) = (\mathbf{Z}^{\mathrm{bi}},\mathbf{Z}^{\mathrm{lat}}).

In other words, the gate is applied only once, and mirroring the input simply swaps the two passes of frame averaging instead of re-splitting the features. No constraint on G\mathbf{G} is therefore needed: invariance and equivariance under the mirror gg are guaranteed by the architecture, whereas similar behavior under a raw electrode mirror is empirical, as shown in our response to the next concern.

Figure 4 tests raw-channel exchange, which Appendix D distinguishes from the learned training transformation, leaving a gap between the architectural construction and the claimed symmetry properties.

For Fig. 4 experiments, if the exchange is applied after the decomposition, you will get the result of +1 vs. -1, as shown in the first two rows below, which is guaranteed by the architecture and thus is uninformative. Therefore, Fig. 4 tests a harder case, exchanging the raw electrodes, which the architecture does not guarantee. 

ExchangeDatasetBilateralLateral
After decompositionEgoBrain+1.00−1.00
After decompositionPhysioNet-MI+1.00−1.00
Raw electrodesEgoBrain+0.98−0.76
Raw electrodesPhysioNet-MI+0.98−0.70

Even under the raw exchange, the bilateral features remain almost unchanged and the lateral features largely flip sign, consistently across all subjects. This shows that the model learns the intended symmetry beyond what the architecture enforces.

We will make the definition of the mirror and the distinction between guaranteed and empirical properties explicit in the revised version.

W2/Q2

The evidence for transferable action-conditioned dynamics is limited: removing prediction has a modest effect in Table 3, shuffling IMU raises the L1 error by only 1–2% in Table 4, and there is no current-state copying baseline or comparably trained action-free predictor. Uncertainty estimates for the action gap are also requested.

We thank the reviewer for the suggestion. We add the two requested baselines to Table 4, a copy baseline and an action-free predictor trained identically but without EEG, and report 95% confidence intervals of Δact\Delta_{\mathrm{act}} from a subject-level paired bootstrap.

ModelWithin L1↓Within Cos.↑Within Δact\Delta_{\mathrm{act}}↑Held-out L1↓Held-out Cos.↑Held-out Δact\Delta_{\mathrm{act}}↑
Random init.0.7334 ± 0.01650.8334+0.0000 ± 0.00000.7224 ± 0.01160.8407+0.0000 ± 0.0000
Copy frame0.5036 ± 0.02340.8783–0.4961 ± 0.01690.8859–
Action-free0.4754 ± 0.02090.8921+0.0044 ± 0.00050.4753 ± 0.01160.8966+0.0021 ± 0.0003
Latent action0.4787 ± 0.02090.8909+0.0026 ± 0.00030.4764 ± 0.01060.8968+0.0015 ± 0.0003
MWM (ours)0.4694 ± 0.02060.8938+0.0085 ± 0.00060.4675 ± 0.01300.8991+0.0071 ± 0.0005

MWM achieves the lowest L1 and a 2–3× larger action gap than the action-free predictor, with non-overlapping confidence intervals on both splits, showing that the gain comes from action conditioning rather than frame prediction alone. The action gap looks small relative to L1 because most of the next frame is already determined by the current frame (see the copy baseline); the correct action accounts for about a quarter of MWM's improvement over copying.

Regarding Table 3, removing prediction costs 5–11% on four tasks (WAYGALReg, Jeong2020, Schwarz, ForenzoFutVec) and at most 3.6% on the other eight, and without prediction the LAT gate collapses (mean 0.85 → 0.04), removing the lateral/bilateral separation.

We will update Table 4 in the revised version.

Q3

What are the exact time intervals of the three IMU summaries relative to the current frame tt and the target t+0.2t+0.2, and do any include measurements after tt?

We thank the reviewer for the question. The three summaries cover [t−0.2,t)[t-0.2,t), [t,t+0.2)[t,t+0.2), and [t+0.2,t+0.4)[t+0.2,t+0.4) s. The middle one is the action of the prediction step itself, i.e., the movement between the current and target frames, following the standard convention of action-conditioned world models that ata_t is executed between sts_t and st+1s_{t+1}. Including measurements after tt (future leakage) is therefore intended: the IMU serves as a proxy for the real action that drives the transition to the target, and this action necessarily takes place after tt. The two neighboring intervals hedge against clock misalignment: the IMU and EEG/video clocks are aligned with a residual of about 0.10 s (std), half a prediction step, so the movement that drives the step can fall into an adjacent interval. Importantly, all compared predictors receive exactly the same IMU inputs, targets, and adapter, so this choice does not favor any model and the comparison remains fair. We will state the intervals explicitly in the revised version.

Thank you for the clarifications and the additional baselines. The updated Table 4 addresses my main concern about whether MWM improves over copying the current frame or pretraining without actions. I have three remaining suggestions.

  1. In Eqs. (6)–(9), please define the mirror on the decomposed feature pair and show how each output transforms. In particular, please specify whether the reported sign flip for lateral features is measured after aligning homologous channels.
  2. In Appendix C, please give the endpoints of all three IMU windows relative to the current and target frames. A comparison that drops only the window after the target frame would show whether MWM retains its advantage when conditioned on the movement leading up to the target.
  3. The rebuttal describes the uncertainty on the action gap as a 95% confidence interval, while Table 4 labels the same values as bootstrap standard deviations. Please reconcile these descriptions and report the subject-level 95% intervals for the action gap.
  4. Could the authors also revise and refine Figures 1–3 to improve their clarity and visual presentation.

The additional results and clarifications have made me more positive about the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.