acceptodds
Withdrawn from ICLR 2027

When Should a World Model Switch? Surprise-Gated State-Space Models for Hierarchical Events

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A predictive world model faces a recurrent dilemma: preserve context while an event is stable, yet rapidly reorganize state when the current dynamics no longer apply. Most video models leave this decision implicit, and most event-boundary detectors mark changes without letting those changes alter the model's internal dynamics. We introduce , a multimodal latent world model that operationalizes Event Segmentation Theory as explicit predictive computation: becomes an endogenous control signal, and a learned hierarchical jointly (i) expands a selective state-space model's discretization step, attenuating stale history, and (ii) retrieves an input-conditioned that initializes the next state. Counterfactual utility regularization teaches the gate to switch only when reorganization improves future prediction, and a learned prior supplies event hazards during open-loop rollout; we prove a monotone bound on the direct contribution of pre-boundary state that a fully open schema-mixture removes exactly. improves controlled open-loop prediction by ( seeds, ) (OOD – in seeds across all four OOD families); on Kinetics-GEBD its internal gate reaches Rel. Dis. mean-F1 (zero-shot 5-seed ensemble on val clips with extracted features; per-seed ) without boundary-supervised representation learning, and the same gate reduces open-loop prediction error on KTH Actions at recurring action-regime starts while remaining a detector at exogenous cuts. In naturalistic fMRI, endogenous innovation adds unique cross-validated predictivity of (; subjects) in a pre-specified event network beyond audiovisual features and raw innovation, replicating on an independent film (, ), while the gate's discrete boundaries dissociably track sensory-change cortex ( subjects). Together, these results establish as a principled mechanism for continuous world modeling, and show that its functional decomposition reflects the organization of the human brain.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, event segmentation, state-space models, Mamba, predictive coding, surprise gating, hierarchical temporal abstraction, fMRI encoding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jfdgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce HEG-VMamba - a multi-modal latent world model that adds a learned event gate to a selective SSM. The NLL of the next latent drives the "hazard" variable. In its turn, the hazard variable enlarges the discretisation step when it fires so that memory decays faster, while mixing the recurrent state toward the learned bank of schemas. The authors report experiments on a synthetic RegimeWorld benchmark, zero-shot boundary detection on Kinetics-GEBD and TAPOS, open-loop prediction on KTH, and encoding analyses of Sherlock fMRI.

Critical strengths
  1. Honestly reported nulls.

  2. Interesting and important problem to study.

Critical weaknesses
  1. Extremely difficult to read and parse.

  2. I think it is important to point out that Table 2 shows that the learned gate does not beat the raw-surprise gate. Furthermore, it shows that the components of the model do not matter that much. For instance, faster forgetting adds very little (-2%); or schema vs. zero-erasure is only -1.7%. What seems to be shown to help is wiping the memory when "surprised" (i.e. a state reset), but this technique is already used in prior work (e.g. HM-RNN).

  3. The authors do not seem to actually evaluate the hierarchy. Section 3.5 specifically states "Hierarchy is evaluated through empirical event duration, autocorrelation, cross-layer nesting, and alignment to independently annotated fine and coarse boundaries.", but there are no results on that? Some other missing results that are claimed to be in the text: (a) section 4.4 says nesting is reported; (b) multimodal attribution of boundaries to visual, speech or cross-modal causes - not found.

  4. Video prediction results on real video are weak against the baselines.

  5. Some citations are missing, for example "Variational Predictive Routing with Nested Subjective Timescales" (ICLR 2022) and related work on event-driven, latent-variable, hierarchical generative models.

  6. In Table 1, HEG has worse 64-step MSE than GateL0RD and THICK, where the caption calls this saturation. Yet the text claims across-the-board superiority. I am not sure I understand why this is the case. Furthermore, the abstract says GEBD is evaluated on 17911 clips, but Table 4 reports on the 8997 test half. In general, the reporting seems inconsistent throughout.

Questions for authors
  1. Is the full model better than the raw-surprise override, the random matched gate, and the no-counterfactual variant, using paired per-seed tests?
  2. Where are the claimed hierarchy results in the paper?
  3. Why are there inconsistencies in the results across the abstract and main body of the paper?
  4. Could the authors be a bit more narrow and precise regarding the claims of the paper? In its current form, the results support the benefit of the surprise-gated state reset, which has previously been shown.
Reviewer_MebuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a video world model that treats prediction failure as a control signal. The model is evaluated on controlled synthetic prediction, natural video, and naturalistic fMRI.

Critical strengths

HEG-VMamba shows improvements over the reported baselines on a controlled regime-shift benchmark and on the “Four-question evidence” graphs. However, because the paper is hard to follow, it is difficult to judge how meaningful these gains are and the significance of these results.

Critical weaknesses

Clarity: The paper is hard to follow. Figure 1, the only architecture diagram, has a one-line caption that doesn’t explain its panels and is cited only once, in passing, at the start of Section 3. The experiments are also poorly explained. Section 4 reports dense strings of statistics without defining key metrics or describing the setup, which is deferred to the appendix, making the results difficult to parse.

The mechanism-identification matrix is insufficient: Table 2’s headline comparison is the full model against a from-scratch no-gate model, and the ablations are one-at-a-time removals rather than a factorial design, so interactions are never tested.

Questions for authors

How does the HEG-VMamba architecture compare to the VMamba architecture? (Liu et al., 2024)

Reviewer_9prTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HEG-VMamba, a multimodal latent world model which augments Mamba with an event-structured switching mechanism, which can decide when to preserve or reorganize predictive states explicitly. In HEG-VMamba, each layer computes calibrated predictive innovation by measuring how much the next input is surprising compared to the model’s predictive uncertainty and this measured innovation is used to compute the event hazard to determine how strongly to switch the model state. Therefore, using this learned event hazard, HEG-VMamba can explicitly determine whether to preserve or reorganize its state by inferring whether an event persists or changes.

To empirically validate the effectiveness of HEG-VMamba, the authors firstly conduct experiments on controlled regime-shift benchmark and HEG-VMamba generally improves open-loop prediction performance compared to other baselines. Also, compared to the no-gate variant, without the event hazard, HEG-VMamba shows clear performance improvements which indicates that explicitly incorporating the event-structured switching mechanism is beneficial under regime shifting scenarios. Also, the authors evaluate HEG-VMamba on natural video datasets. In Kinetics-GEBD and TAPOS tasks, the authors show that learned event hazard can capture event boundaries without boundary-supervised representation learning. Also, in KTH Actions task, HEG-VMamba shows better open-loop prediction at action transitions. Finally, the authors evaluate HEG-VMamba in naturalistic movie-fMRI recordings and suggest that predictive innovation and the learned event hazard track distinct brain regions.

Critical strengths

Well-motivated problem formulation and well-designed experimental controls : I think the authors raise a reasonable point that the stability–plasticity problem over time should be treated in continuous world modeling. Since our real-world experience can be viewed as a sequence of events, it is natural for a model to preserve its state within a single event but reorganize the state when the event changes. From this intuition, the architectural design of HEG-VMamba seems also natural which augments Mamba with a learned event hazard module to determine whether to preserve or reorganize the model states. Also, to experimentally validate the effectiveness of this design choice, the authors conduct well-designed ablation studies. Especially, they compare HEG-VMamba with various variants to identify which components actually contribute to the empirical performance gain in HEG-VMamba. Overall, the authors formulate a well-motivated problem, proposes a natural architectural design and validates it under reasonable settings, so I think the overall logical flow is clear and well-written.

Critical weaknesses
  1. Limited methodological novelty : Although the architectural design of HEG-VMamba is natural and well-motivated, I think its methodological novelty is somewhat unclear since it seems that most of its individual components are adopted from prior works. For example, as the authors note in the paper, leveraging prediction error to determine whether to change the model states has already been explored in other models such as SUGAR and SUNTA. The authors position their contribution as directly coupling the gate (event hazard module) to the SSM discretization and schema retrieval, however, in my opinion, this coupling itself does not seem to be a fundamentally new approach since the original Mamba already includes an input-dependent discretization step to control how much of the previous state is retained and retrieving an input-conditioned mixture of learned schema seems closely related to the idea of switching state-space models. Therefore, I think a more thorough discussion is needed to clarify the core methodological contributions of HEG-VMamba.

  2. Limited empirical gains : In the controlled synthetic benchmark, HEG-VMamba generally shows performance improvements in open-loop prediction, but it seems that its predictive gains on natural videos are limited. As I understand, on natural videos, gate-specific prediction improvement is reported only on KTH Actions, where the improvement is observed at action-regime starts but not at intra-action periodic phases. Also, as the authors note in the paper, the gate does not lower open-loop error on Kinetics-GEBD and its gate-specific contribution is approximately zero in the long-form analysis. Since HEG-VMamba is proposed as a world model, in my opinion, its predictive gains should be demonstrated stronger, beyond the controlled synthetic benchmark.

Questions for authors
  1. Clarification of methodological novelty : As described above, could the authors clarify the core methodological novelty of HEG-VMamba compared to prior works such as SUGAR, SUNTA, the original Mamba, and switching state-space models? Especially, I wonder what specific advantage coupling the event hazard to the SSM discretization and schema retrieval enables compared to the other methods. Also, I wonder whether the learned event hazard provides a clear benefit over a simple surprise-triggered reset. Since in Table 2, it seems that replacing the learned event hazard with raw surprise performs comparably to the full model, so it would be helpful if the authors could clarify how much the learned event hazard itself contributes to the performance gain.

  2. Further evidence of predictive gains on real-world settings : Could the authors provide more clear evidence of the predictive gains of HEG-VMamba in real-world settings beyond the KTH Actions results? As I understand, the prediction improvement on natural videos is currently reported only at action-regime starts on KTH Actions, therefore I wonder whether the event hazard also contributes to performance gains in any other real-world settings. If such gains are limited, it would be helpful if the authors could discuss event switching improves prediction on KTH Actions but limited in other natural video settings.

  3. Neuroscientific interpretation of the fMRI experiments : I think the fMRI experiment is a very interesting real-world application. Although the authors provide some interpretations about the observed dissociation, however, I wonder whether the authors could provide a more detailed neuroscientific interpretation of this dissociation. I believe that if such an interpretation contributes to a deeper understanding of how the human brain processes events, it would be a clear and significant contribution of this work. However, I understand that providing such valuable real-world contributions is challenging, so this point will not negatively affect my evaluation even if it is not fully explained.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.