- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper asks what information a latent state should keep so that the future of a multimodal social interaction (language, audio, video) can be predicted from that state alone. The authors call this "predictive sufficiency": once the state and the current event are known, the rest of the history should add little about the future.
To learn such a state, PSM-WM combines four parts. The state is a Gaussian belief regularized toward N(0, I). The state is trained to predict the output of a pretrained, frozen future encoder. An event-conditioned mixture-of-Gaussians model learns how the state moves forward in time. A multi-step rollout loss keeps long-horizon predictions consistent.
The method is validated with future-prediction W-F1 on Hi-EF, IEMOCAP, and MELD against multimodal, predictive-representation, and world-model baselines. The paper also proposes three probe metrics on the frozen state (Predictive Closure Gap, Residual History Gain, and Future Latent Retrieval). It adds ablations, missing-modality and calibration tests, and a transfer experiment on Ego4D.
Critical strengths
S1. The paper is well written and easy to read through. The introduction motivates the problem well. Fig. 1 shows that two situations with the same current semantics can lead to different futures. This makes the case for a future-oriented state easy to follow.
S2. The paper separates two questions that world models often mix: what a latent state should keep, and how that state should evolve. Stating the goal as p(F | H, E) ≈ p(F | S, E) gives a clear target. That target is relevant beyond social interaction modeling.
S3. Evaluating the state, not only the forecast. The paper does not stop at forecasting accuracy. It also tries to test the learned state directly with frozen-representation probes (PCG, RHG, FLR). It adds robustness to missing modalities, uncertainty calibration, and an out-of-domain test on Ego4D. This kind of state-level evaluation is a useful direction for the field, even though I have concerns about how it is carried out (see weaknesses).
Critical weaknesses
W1. Novelty of the method. The introduction motivation is strong, but the method is not. Predictively Sufficient Multimodal Social World Modeling sounds very similar to causal modeling, but the paper is just adding a few regularizers to future prediction. I do not believe it solves a novel issue in the way the introduction and motivation say. Using a predictive state as a latent variable is what most predictive modeling already does. The added claim is "reducing redundant dependence on the observed history". But nothing in the method actually addresses that redundancy, and the state does not become explainable.
W2. Missing standard deviations / significance. Table 1 has no standard deviations. The model is a stochastic dynamics model, as the paper itself claims. So it is even more important to report standard deviations for the method and the baselines, to show whether it really has an advantage. L269 says standard deviations are reported, but they are not in the main results. Fig. 3(b) also looks scaled so that minor differences look bigger. It needs a standard-deviation spread to show whether the improvement is meaningful.
Questions for authors
Q1. Frozen future encoder (Fig. 2, Sec. 2.3).
- Is the information in the latent limited by how well this frozen encoder encodes the future?
- Is it ever trained together with the other objectives, especially the predictive sufficiency objective, either jointly or in stages?
- How is the frozen future encoder trained? As an autoencoder?
- Fig. 3 has a bar for a "generic encoder". What is it? Is it the frozen future encoder from Fig. 2, or a different one?
- How do you make sure that the frozen future encoder keeps the predictively sufficient features that are used as targets?
Q2. Fig. 3(c) shows a long horizon, but the x-axis only goes 1-2-3. Is 3 the longest horizon? Section 3.2 (experimental setup) does not give this detail.
Q3. L86-87 talks about social interactions (Fig. 2). In practice that means surveillance videos or robot interactions, which usually have audio and video only. Text would have to be transcribed by a separate model. Movies are different because their text is annotated. Real-world use cases do not come with annotated text, so does the method depend on a separate transcription model?
Minor / presentation: Q4. E_t is used at L80 and L82 but is not defined before that. It is first defined at L188.