- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents CLWM, a cognitive liquid world model that reformulates physical world modeling from passive video as continuous-time causal hypothesis testing over decoupled latent representations. On top of frozen visual and text encoders, the authors train a lightweight video-text verification model combining a dual-stream liquid network, codebook memory, and query-conditioned verification. Stage one builds asymmetric semantic and motion streams from pretrained features and predicts the next feature state via self-supervision. Stage two uses LLM-generated questions, relevance-based modulation, and TGPO to produce three-way verification decisions. Evaluation is on Something-Something-v2.
Critical strengths
The paper tackles a reasonable weakness in video models—their reliance on static appearance at the expense of temporal dynamics. The proposed dual-stream design with an adjustable temporal update mechanism is a sensible direction with clear value.
By reusing frozen features from pretrained encoders, the method keeps downstream learning focused on dynamics and verification. The authors report that two-stage training runs on a single RTX 5090, which shows the setup is practically feasible.
Critical weaknesses
Feature bypass and implementation inconsistencies undermine the core method: Equation (5) queries the codebook directly with the original 768-dimensional features, letting information bypass the 32-dimensional hidden state—so the bottleneck alone can't justify the claim that appearance information is eliminated. Meanwhile, Algorithm 3 fixes the clock multiplier at γ = 4 and computes the visual trajectory only once per video, which directly conflicts with the query-dependent clock modulation described in the main text and Table E.1. This leaves the actual implementation of a central mechanism ambiguous.
No external baselines for evaluation: While the introduction draws conceptual comparisons to RL world models, generative models, V-JEPA, and standard liquid networks like LTC/CfC, the paper provides no quantitative results against any external baseline. Since CLWM depends on three frozen pretrained backbones, the lack of even simple baselines using the same features makes it impossible to tell whether performance gains come from the proposed liquid dynamics or from the upstream representations.
Questions for authors
Equation (5) concatenates the original 768-dimensional features with the hidden state. Why wouldn't this path retain appearance information? What if you queried the codebook using only the 32-dimensional hidden state?
Algorithm 3 fixes γ = 4 and computes the visual trajectory once per video, yet Table E.1 credits the gains to query-dependent clock modulation. Which version was actually used in these experiments?
Please add baselines—e.g., GRU or MLP models using the same frozen features—alongside comparisons to other relevant methods to shore up the empirical evidence.
In Stage 2, what information goes into the video descriptions fed to the text-only Qwen model for generating questions about visual properties like speed and contact boundaries, and which annotations supply that information? Please show concrete examples of videos, input descriptions, generated questions, and explain how you verify factual consistency with the video, answerability, and specificity to the individual clip.
Why do the same full model, evaluated on the same asymmetric test set at the same threshold, yield different results across Tables 2/E.2, 3, D.1, and E.3? The ROC-AUC values are 0.9592 / 0.9597 / 0.9784 / 0.9669, and TNR values are 87.00% / 86.71% / 96.30% / 91.62%.
Thank you for your thorough technical audit. We resolved all five points directly in our revision:
Response to Issue 1 & Question 1
In Stage 1, Eq. (5) queries concatenated inputs and states solely to reconstruct full-rank 768d targets (ŝ(t+1), k̂(t+1)), sculpting ODE attractor basins. In Stage 2, we detached the CPC readout completely (0d raw bypass, Eq. 10; Figure 1; Table 2) so the verifier receives only the 128d ventral and 32d dorsal ODE trajectories hv(1:T) and hd(1:T). Removing raw 768d VideoMAE features purges static clutter, lifting canonical asymmetric performance from 0.9592 to 0.9724 ROC-AUC (90.78% Acc, 88.24% TPR, 93.27% TNR at θ = 0.60 vs. 88.81% / 86.54% / 91.08% for Eq. 5; 0.9888 pooled / 0.9946 macro ROC-AUC across the 20-trigger taxonomy in Table D.2, including 0.9627 on surface adhesion/attachment). Zeroing the 32d dorsal (0.5349 AUC) or ventral (0.5304 AUC) stream collapses discriminability to chance (Table 2), confirming zero 768d leakage.
Response to Issue 1 & Question 2
All reported benchmarks (Section 4.3 CTEM, Table E.1) used the fully conditioned dynamic loop, expanding each video across candidate queries ([174, T, D]) so each query modulates τ(eff) = τ · clamp(4.0 - 3.5 · R(t), 0.5, 4.0). Original Algorithm 3 inadvertently transcribed our fast-inference cache (fixed γ = 4.0); we updated Algorithm 3 (Appendix C.4) to formalize the batched dynamic pass and noted single-pass caching in the remark.
Response to Issue 2 & Question 3
Action-conditioned models (DreamerV3, IRIS) require motor actions absent in passive SSv2 video, while pixel generators (Sora, Genie) optimize RGB synthesis rather than causal verification (Table A.2). We trained five parameter-matched baselines (Bi-RNN, Bi-GRU, Bi-LSTM, Temporal MLP, Transformer) for 25 epochs (~98k steps) on identical 1280d frozen features, splits, and TGPO loss (Section 4.2, Table 3; Table E.4). Unified baselines reach 0.9620–0.9801 clean AUC, yet Transformer and Bi-GRU affirm 47.23% and 27.90% of scrambled clips, and all five retain 0.7280–0.8226 V-KO AUC via static texture leakage in 768d motion tokens—whereas CLWM achieves 90.78% clean accuracy, 0.11% scrambled TPR, and 0.5304 V-KO AUC.
Response to Question 4
Qwen2.5-7B-Instruct never infers kinematics from pixels; it reformulates human-annotated, object-grounded SSv2 captions (e.g., "Pushing [something] from left to right" + "ceramic mug" → "Pushing a ceramic mug from left to right") strictly as a syntactic formatter (Appendix D.2, Table D.1):
Triplets pass automated checks for JSON validity, verbatim retention of annotated SSv2 object nouns in semantic/binary queries (specificity and answerability), and non-degeneracy; negatives pair queries with time-reversed clips (Flip) or same-noun distractors (pushing vs. pulling the same mug).
Response to Question 5
We re-evaluated all tables on the Strict 32d ODE checkpoint and annotated jitter, threshold θ, and negative sampling in every caption:
†Table 2 uses θ = 0.60 (91.08% vs. 93.27% TNR); Table E.2 uses θ = 0.40 (87.00% vs. 90.53% TNR, 0.9719 AUC on an independent jitter draw) and reports θ in {0.40, 0.50, 0.60}.