- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper extends recognition-parameterized models (RPMs) to multimodal timeseries in order to learn shared latent dynamics without reconstructing observations. A shared latent evolves on the sphere: modality-specific encoders map observations to von Mises-Fisher recognition potentials, evidence is fused by adding natural parameters (a product-of-experts / assumed-density filter whose gain discounts past evidence), and the latent dynamics are a learned orthogonal rotation plus vMF diffusion. Because vMF factors remain log-linear in the latent even with observation-dependent concentration, fusion and contrastive logits stay analytic. The authors derive a CLIP-like minibatch objective argued to approximate the RPM ELBO, consisting of a candidate-based recognition reward plus an analytic KL to the transition prior. In synthetic experiments the minibatch objective tracks a reference ELBO estimate and the model gives the lowest dynamics-recovery error, lying near an oracle Pareto frontier. On mouse LFP plus 3D pose it improves cross-modal retrieval and LFP-to-keypoint decoding, and yields rearing-related structure.
Critical strengths
- The model is principled and unifies contrastive representation learning with explicit shared dynamics. The vMF construction simultaneously supports additive evidence fusion, an analytic transition KL and a uniform stationary prior, giving an elegant answer to how to infer one latent state from either or both modalities without an observation model. This is consequential because it cleanly separates reconstruction-driven private-variation bias from the shared-dynamics objective.
- The paper gives a convincing mechanistic diagnosis of when joint learning matters. The demonstration that static contrastive alignment recovers instantaneous structure but loses weakly informative yet persistently predictive signal, and that the no-transition ablation fails only when the recurrent filter is also removed, provides a concrete account of the benefit of dynamics-informed representation rather than a leaderboard gain.
- The spherical-geometry argument is well developed. Appendix D.1 carefully shows why a first-order critic requires constant precision in the Euclidean case and how fixed radius removes the radial curvature, which is a genuinely useful conceptual contribution.
Critical weaknesses
- The probabilistic justification is weaker than the framing. A finite, non-decomposable minibatch objective with a candidate-estimated normalizer does not by itself give a valid ELBO; the claim that training optimizes an approximation to the ELBO needs clearer caveats and, ideally, an analysis of the approximation gap.
- The empirical advantage is small and the strongest baseline is often close. In retrieval, sRP-SSM (35%) versus CLIP-posthoc (33%) is within the reported error; for keypoint-to-LFP decoding CLIP-posthoc actually wins ( 0.166 versus 0.124). With a single animal and no cross-animal validation, the practical case for joint dynamics is not established.
- The ablation weakens the role of the stochastic dynamics. Since the transition term and backpropagation through the filter are largely redundant, and the no-transition model matches the reference when the filter is present, it is unclear how much of the benefit comes from the explicit transition model rather than from the recurrent inductive bias.
Questions for authors
- Can you bound or at least empirically quantify the gap between the minibatch training objective (finite candidates and a minibatch normalizer) and the true RPM ELBO? Is the reported tracking a valid bound, or does it also absorb the optimization path?
- Given that CLIP-posthoc matches or beats sRP-SSM in several settings, what is the minimal regime (noise level, private-variation magnitude, horizon) in which jointly learning dynamics is decisively worth it?
- Why is the transition term redundant with the filter in the ablation, and does that redundancy persist on the real data? If the filter alone supplies the inductive bias, is the stochastic dynamics model doing independent work?
- The real-data results come from one mouse. How do you expect the retrieval and decoding advantages to generalize, and can you report multi-animal variability? If not, some discussions are welcome I think.
We thank the reviewer for the detailed and constructive review.
Quantify the ELBO approximation: You are correct that the finite minibatch objective is not guaranteed to lower bound the log evidence. The objective differs from the RPM ELBO in two ways. First, it replaces the full normalizer with a finite candidate set (Fϕ vs. F^ϕ). This error can have either sign and vanishes as the candidate estimate converges to the full normalizer. Second, it applies a Jensen bound to the finite-candidate objective (Eq. 11). This second step is guaranteed to be a lower bound to the finite-candidate objective.
On synthetic data, we estimated the ELBO using a 100,000 candidate reference bank and Monte Carlo posterior expectations as in Figure 2B, but now tracking estimates of both the finite-candidate and Jensen contributions to the minibatch vs. reference ELBO discrepancy. The minibatch objective was below the reference at all 90 checkpoints across five seeds. At the end of training, the total gap was −0.047±0.001 nats per timestep, with at most 0.01 nats attributable to the Jensen step. We evaluated both objectives at each checkpoint on the same held-out windows with a fixed candidate bank and posterior sampling seed. The reported correlation of 0.9985 therefore measures tracking under a fixed evaluation protocol along the optimization trajectories, while the measured gaps quantify the approximation error. In the revision we will highlight the two sources of RPM ELBO discrepancy throughout training.
Transition term vs. filter ablation Following your suggestion, we performed the same 2×2 ablation on real data. We find that joint learning of dynamics via the transition term improves over training with a filter alone in retrieval, latent predictability, and keypoint decoding. With dynamics refit at evaluation, adding the transition loss improved top-1 retrieval by 5.4±0.2 percentage points without the training filter and by 1.1±0.5 points with it (paired across three seeds). The smaller gain with the filter active suggests that the two training signals partly substitute for each other. Please see our response to reviewer WK4L for full details.
When joint learning helps. Across four mice, sRP-SSM and CLIP-posthoc had similar zero-lag retrieval, while sRP-SSM had a longer integrated timescale for future matching by 0.13 - 0.41 s. Within sRP-SSM, training with both the recurrent filter and transition loss increased this timescale by 0.149±0.031 s relative to training with neither (paired across three seeds, with dynamics refit at evaluation). These results point to a benefit for matching at future lags. The minimum noise, private variation, or temporal persistence needed for that benefit remains open.
Additional mice: Following your suggestion, we repeated the neural–behavioral experiments in three additional mice, fitting models separately for each animal. Across all four mice, both CLIP-posthoc and sRP-SSM substantially improved cross-modal retrieval over CLIP. The sRP-SSM advantage over CLIP-posthoc varied across mice, with differences ranging from −3.5 to +2.9 percentage points.
The clearest advantage of sRP-SSM was in future cross-modal matching. CLIP-posthoc and sRP-SSM start with a similar zero-lag accuracy, but sRP-SSM is able to match substantially further into the future (0.13-0.41 seconds further across mice as measured by integrated timescale). sRP-SSM also achieved the highest mean keypoint R2 for LFP-to-pose decoding in all four mice. For pose-to-LFP log spectrum decoding, CLIP-posthoc had the highest average R2 in three of four mice.
Additionally, the regional pattern of LFP log spectrum decoding was consistent across mice (Figure 3F). In mice, pose predicted 30-60 Hz power most strongly in the amygdala, the ventral hippocampus, and the nucleus accumbens (R2=0.13−−0.28). The full region-by-frequency maps correlated with the original mouse's map at r=0.63 to 0.81.