- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces CardiacState, a joint-embedding predictive architecture for echocardiograms. It infers an explicit cyclic state on from visual history, and combines this state with current-frame features and physical elapsed time to predict future representations. A causal temporal bottleneck restricts the future predictor from directly accessing historical frames, forcing temporal context through the low-dimensional cyclic state.
Core Claim: In action-free dynamic systems, combining a structured low-dimensional state with an information bottleneck enables predictive models to learn physically grounded endogenous dynamic states directly from visual streams.
Critical strengths
-
Well-motivated endogenous state formulation: Unlike action-conditioned world models that rely on external actions, cardiac motion lacks external control signals. CardiacState infers an internal state and constrains it to an manifold. Importantly, this cyclic state is not just a passive auxiliary loss; it actively conditions future feature prediction.
-
Information bottleneck as a general representation principle: By denying the predictor direct access to historical visual tokens (), historical dynamics are forced through the explicit state channel. In Table 3, progression regularization drops localization MAE from 18.396 to 2.107 frames, and enforcing further reduces it to 2.002 frames while improving frozen LVEF MAE from 5.028% to 4.867%. These results support combining structured states with temporal bottlenecks.
Critical weaknesses
-
Cyclic states are not novel, and the impact of the bottleneck is relatively marginal: Using cyclic representations for cardiac cycles has been explored in prior work (e.g., DeepHeartBeat, OrbitalEcho). The primary novelty here is using as a condition under an architectural bottleneck. However, the ablations in Table 3 show that the dominant gain comes from the progression loss (MAE 18.396 2.107), whereas reducing context from to yields a much smaller increment (2.107 2.002). This indicates the main improvement stems from explicit temporal regularization rather than the novel bottleneck design itself.
-
The bottleneck does not guarantee state utilization: Restricting direct frame access forces history through the state pathway if historical information is used, but it does not prove the predictor relies on this channel. The model could still rely primarily on the current frame feature and elapsed time , while the cyclic state organizes independently under . The experiments do not fully rule out this alternative explanation. Additionally, keyframe localization evaluates error against the nearest predicted event; increasing the number of predicted events can naturally lower this distance without necessarily reflecting better physiological phase alignment.
Questions for authors
-
Under fixed , what happens to future feature prediction and downstream LVEF if the cyclic state is ablated, shuffled across time, or replaced with a fixed periodic signal? Does the predictor strictly rely on the state inferred from visual history?
-
Does the learned cyclic state reliably correspond to true cardiac cycles? After controlling for the number of predicted events and adopting a more stringent event-matching protocol, does the advantage in keyframe localization still hold?
7 comments
2. Direct Proof that the Predictor Relies on the Inferred State (W2 & Q1)
For downstream LVEF estimation, we evaluate the frozen visual encoder with the state module and predictor detached; the state pathway is trained jointly with during pretraining (and used explicitly for ED/ES localization), so its effect on LVEF is mediated through the enriched visual features (removing the state module during pretraining degrades Dynamic LVEF MAE from to ).
To directly test whether the next-feature predictor relies on the inferred state rather than ignoring it in favor of , we froze each pretrained model and rotated the current cardiac state at inference while keeping all other inputs fixed. Below we report the feature RMSE between predictions made with the rotated state and those made with the original state across all three datasets:
EchoNet-Dynamic
| Rotation | , w/o | , w/ | , w/ |
|---|---|---|---|
EchoNet-Pediatric
| Rotation | , w/o | , w/ | , w/ |
|---|---|---|---|
EchoNet-LVH
| Rotation | , w/o | , w/ | , w/ |
|---|---|---|---|
The predicted features diverge systematically from the original predictions as the rotation angle increases, and the changes are larger at than at across all tested angles and datasets. This directly confirms that the predictor actively uses the cardiac state and relies on it substantially more under the tighter bottleneck.
3. Controlling for Detection Count in Keyframe Localization (W2 & Q2)
We moved Figure 2 from the supplement into the main paper, illustrating that the state learned by the model with follows true cardiac cycles across beats in the representative video.
In the submitted evaluation, all CardiacState variants used the same 5-frame local temporal window to suppress nearby candidate detections during ED/ES peak selection (Appendix C). To assess detection counts under sparse annotations, we estimate each video's cycle count by dividing its length in frames by twice the annotated ED–ES interval. Detection density is the number of predicted ED and ES events divided by the estimated cycle count.
\text{detection density} = \frac{N_{\text{predictions}}}{\widehat{N}_{\text{cycles}}}.$$ One ED and one ES event per beat corresponds to an expected density of approximately $2.0$. Across every acoustic view, our $K=1$ model averages **$2.280 \pm 0.934$** detections per estimated cycle and achieves lower localization MAE than $K=4$ while generating **fewer** detections per estimated cycle—ruling out over-detection as the source of localization accuracy. A beat-by-beat assessment under a one-to-one event-matching protocol would require exhaustive multi-cycle ED/ES annotations, which we note in Section 5 as an important direction for future work. | View | Method | MAE $\downarrow$ | RMSE $\downarrow$ | $R^2$ $\uparrow$ | Estimated cycles | Estimated cycle frame length | Estimated detection density | | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | A4C | $K=4$, w/o $\mathcal{L}_{\text{cardiac}}$ | $28.475 \pm 0.997$ | $49.082 \pm 1.395$ | $-0.429 \pm 0.094$ | $5.50 \pm 3.13$ | $33.91 \pm 10.98$ | $2.217 \pm 1.979$ | | A4C | $K=4$, w/ $\mathcal{L}_{\text{cardiac}}$ | $2.105 \pm 0.029$ | $2.518 \pm 0.028$ | $0.9962 \pm 0.0002$ | $5.50 \pm 3.13$ | $33.91 \pm 10.98$ | $2.200 \pm 0.708$ | | A4C | $K=1$, w/ $\mathcal{L}_{\text{cardiac}}$ | **$1.985 \pm 0.024$** | **$2.325 \pm 0.022$** | **$0.9967 \pm 0.0002$** | $5.50 \pm 3.13$ | $33.91 \pm 10.98$ | **$2.137 \pm 0.692$** | | PSAX | $K=4$, w/o $\mathcal{L}_{\text{cardiac}}$ | $15.183 \pm 0.883$ | $27.059 \pm 1.690$ | $0.501 \pm 0.137$ | $2.67 \pm 1.67$ | $39.42 \pm 21.30$ | $2.724 \pm 2.971$ | | PSAX | $K=4$, w/ $\mathcal{L}_{\text{cardiac}}$ | $2.083 \pm 0.047$ | $2.573 \pm 0.051$ | $0.9953 \pm 0.0011$ | $2.67 \pm 1.67$ | $39.42 \pm 21.30$ | $2.546 \pm 1.370$ | | PSAX | $K=1$, w/ $\mathcal{L}_{\text{cardiac}}$ | **$2.027 \pm 0.045$** | **$2.452 \pm 0.054$** | **$0.9958 \pm 0.0010$** | $2.67 \pm 1.67$ | $39.42 \pm 21.30$ | **$2.518 \pm 1.332$** | | PLAX | $K=4$, w/o $\mathcal{L}_{\text{cardiac}}$ | $11.531 \pm 1.055$ | $24.956 \pm 2.077$ | $0.572 \pm 0.101$ | $5.20 \pm 1.83$ | $34.68 \pm 8.29$ | $3.213 \pm 1.558$ | | PLAX | $K=4$, w/ $\mathcal{L}_{\text{cardiac}}$ | $2.135 \pm 0.061$ | $2.597 \pm 0.058$ | $0.9954 \pm 0.0008$ | $5.20 \pm 1.83$ | $34.68 \pm 8.29$ | $2.327 \pm 0.562$ | | PLAX | $K=1$, w/ $\mathcal{L}_{\text{cardiac}}$ | **$1.994 \pm 0.049$** | **$2.358 \pm 0.046$** | **$0.9962 \pm 0.0006$** | $5.20 \pm 1.83$ | $34.68 \pm 8.29$ | **$2.185 \pm 0.528$** | | Avg | $K=4$, w/o $\mathcal{L}_{\text{cardiac}}$ | $18.396 \pm 0.570$ | $33.699 \pm 1.011$ | $0.215 \pm 0.065$ | $4.46 \pm 2.63$ | $36.01 \pm 14.84$ | $2.718 \pm 2.285$ | | Avg | $K=4$, w/ $\mathcal{L}_{\text{cardiac}}$ | $2.107 \pm 0.027$ | $2.563 \pm 0.028$ | $0.9956 \pm 0.0005$ | $4.46 \pm 2.63$ | $36.01 \pm 14.84$ | $2.357 \pm 0.958$ | | Avg | $K=1$, w/ $\mathcal{L}_{\text{cardiac}}$ | **$2.002 \pm 0.023$** | **$2.378 \pm 0.024$** | **$0.9962 \pm 0.0004$** | $4.46 \pm 2.63$ | $36.01 \pm 14.84$ | **$2.280 \pm 0.934$** |
We thank Reviewer 3 for the sharp, focused questions regarding the contribution of the K=1 bottleneck, causal state utilization by the predictor, and event-count control in keyframe evaluation.
1. Value of the K=1 Temporal Bottleneck Beyond Lcardiac (
W1)We thank the reviewer for raising this important distinction. To isolate the contributions of explicit cyclic-state conditioning and the causal bottleneck from standard feature prediction, we trained a matched time-conditioned K=4 JEPA baseline without the cardiac-state module or Lcardiac under identical initialization, pretraining data, sampling, and compute budget. Because this baseline produces no phase representation, keyframe localization is not applicable (
N/A), and we evaluate its frozen visual representations on LVEF estimation alongside our ablation variants:N/A). Making cyclic conditioning work in a predictive architecture requires coupling S1 with our echocardiography-grounded inductive bias (Lcardiac, which enforces physical-time-scaled forward progression while accommodating heart-rate-dependent subphase velocities) and the K=1 bottleneck.