- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce HEG-VMamba - a multi-modal latent world model that adds a learned event gate to a selective SSM. The NLL of the next latent drives the "hazard" variable. In its turn, the hazard variable enlarges the discretisation step when it fires so that memory decays faster, while mixing the recurrent state toward the learned bank of schemas. The authors report experiments on a synthetic RegimeWorld benchmark, zero-shot boundary detection on Kinetics-GEBD and TAPOS, open-loop prediction on KTH, and encoding analyses of Sherlock fMRI.
Critical strengths
-
Honestly reported nulls.
-
Interesting and important problem to study.
Critical weaknesses
-
Extremely difficult to read and parse.
-
I think it is important to point out that Table 2 shows that the learned gate does not beat the raw-surprise gate. Furthermore, it shows that the components of the model do not matter that much. For instance, faster forgetting adds very little (-2%); or schema vs. zero-erasure is only -1.7%. What seems to be shown to help is wiping the memory when "surprised" (i.e. a state reset), but this technique is already used in prior work (e.g. HM-RNN).
-
The authors do not seem to actually evaluate the hierarchy. Section 3.5 specifically states "Hierarchy is evaluated through empirical event duration, autocorrelation, cross-layer nesting, and alignment to independently annotated fine and coarse boundaries.", but there are no results on that? Some other missing results that are claimed to be in the text: (a) section 4.4 says nesting is reported; (b) multimodal attribution of boundaries to visual, speech or cross-modal causes - not found.
-
Video prediction results on real video are weak against the baselines.
-
Some citations are missing, for example "Variational Predictive Routing with Nested Subjective Timescales" (ICLR 2022) and related work on event-driven, latent-variable, hierarchical generative models.
-
In Table 1, HEG has worse 64-step MSE than GateL0RD and THICK, where the caption calls this saturation. Yet the text claims across-the-board superiority. I am not sure I understand why this is the case. Furthermore, the abstract says GEBD is evaluated on 17911 clips, but Table 4 reports on the 8997 test half. In general, the reporting seems inconsistent throughout.
Questions for authors
- Is the full model better than the raw-surprise override, the random matched gate, and the no-counterfactual variant, using paired per-seed tests?
- Where are the claimed hierarchy results in the paper?
- Why are there inconsistencies in the results across the abstract and main body of the paper?
- Could the authors be a bit more narrow and precise regarding the claims of the paper? In its current form, the results support the benefit of the surprise-gated state reset, which has previously been shown.