acceptodds
Under review as a conference paper at ICLR 2027

LIFT-JEPA: Task-Aware Joint-Embedding Supervision via Target Interventions for Spatiotemporal Forecasting

Abstract

In spatiotemporal forecasting where task-critical phenomena are sparse, prediction losses applied in observation space allow pixel-wise errors in task-critical regions to be directly reweighted. By contrast, joint-embedding predictive learning applies the loss in representation space; because latent units lack an explicit correspondence to task semantics expressed in observation space, observation-space task weights cannot be directly applied to the representation-matching loss. To this end, we introduce LIFT-JEPA (Latent Importance From Target Interventions for Joint-Embedding Prediction). Given a future observation, LIFT-JEPA constructs an intervened counterpart by removing task-critical structures and encodes both using the same target encoder. The resulting representation differences are used to estimate the task relevance of individual representation units and reweight their matching losses, thereby translating task priorities specified in observation space into representation-space supervision. We evaluate LIFT-JEPA across precipitation nowcasting and urban traffic flow prediction using three forecasting backbones with distinct inductive biases. Compared with Vanilla JEPA and alternative task-weighting strategies, LIFT-JEPA consistently improves task-relevant forecasting performance across both domains, from intensity-defined targets in radar nowcasting to temporal-drift targets in traffic flow prediction. These results demonstrate that LIFT-JEPA can translate task semantics ranging from spatial intensity structures to temporal drift into effective representation-space supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
joint-embedding predictive architectures, self-supervised representation learning, task-aware supervision, spatiotemporal forecasting, radar precipitation nowcasting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YvEFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

(There may be inadvertent errors in any areas where AI is not used, such as in writing or reading academic papers.)

This paper is the first to attempt a JEPA framework in the video prediction area.

Basically, this framework consists of an encoder-predictor-decoder architecture. The encoder encodes observation data, and then the predictor predicts future feature states, and the decoder decodes them into observation-domain data.

Because the observation domain has a lot of risks (e.g., noise structure), attempting a JEPA framework is reasonable.

Critical strengths

In this paper competitively provide detailed evaluation protocols

Also, that paper evaluated a lot of dataset like Shanghai Radar, MeteoNet.

Critical weaknesses

The two main weaknesses are evaluation and novelty.

First, regarding evaluation, SEVIR is a standard dataset for weather prediction, but the authors chose two less commonly used datasets, making comparisons with other methods difficult.

For example, SEVIR is essential for comparisons with models such as RainODE [1], CasCast [2], and PreDiff [3]. Even if this paper does not directly compare against these models, results on SEVIR are necessary for readers to make their own comparisons.

Also, on the TrafficBJ traffic prediction dataset, the reported baseline results are worse than those in the official OpenSTL [4] repository. The authors might argue that they could not reproduce those results in their own experiments. However, OpenSTL is well known for its reproducibility and is widely used by many researchers, including myself. Therefore, the authors should properly reproduce baselines such as SimVP and TAU using the OpenSTL framework.

Regarding novelty, although this method applies the JEPA framework to video prediction, SimVP already follows the same concept: an encoder compresses observations into features, a predictor transforms these features into future states, and a decoder reconstructs the observations. Therefore, the method feels like it mainly adds stabilization techniques such as EMA and makes training more complicated.

[1] Seong, Yeeun, et al. "RainODE: Continuous-Time Precipitation Forecasting with Latent Neural ODEs." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[2] Gong, Junchao, et al. "Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling." arXiv preprint arXiv:2402.04290 (2024).

[3] Gao, Zhihan, et al. "Prediff: Precipitation nowcasting with latent diffusion models." Advances in Neural Information Processing Systems 36 (2023): 78621-78656.

[4] https://github.com/chengtan9907/OpenSTL/blob/OpenSTL-Lightning/docs/en/model_zoos/traffic_benchmarks.md

Questions for authors

N/A

Reviewer_oW4VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Motivated by the sparsity of task-critical structures and existing methods’ emphasis on addressing this imbalance in representation space, this paper proposes LIFT-JEPA, a task-aware approach based on observation-space interventions. It compares representations of the original future and a version with selected structures removed, using their differences to weight representation matching toward the original target. The method is evaluated across three forecasting backbones and three datasets.

Critical strengths

S1. The paper is motivated by a reasonable observation: sparse, task-critical structures may receive insufficient emphasis under uniform representation matching. The presentation is clear, and the method and training procedure are easy to follow.

S2. The evaluation covers three representative forecasting backbones—SimVP, PhyDNet, and TAU—across radar and traffic forecasting, providing evidence that the approach is applicable to different architectures and task priorities.

S3. The controlled ablations are well designed. Comparisons with random masks, shuffled weights, and directly pooled masks help distinguish the effects of task-specific region selection and representation-dependent weighting, providing useful empirical support for the proposed mechanism.

Critical weaknesses

W1. Although LIFT-JEPA improves the targeted metrics on average, these gains involve trade-offs: radar FAR increases in all settings, and traffic Global MAE consistently worsens. SSIM gains are very small relative to the reported standard deviations. Moreover, some target-metric gains, particularly for PhyDNet on TaxiBJ, are modest relative to run-to-run variability, leaving their robustness unclear.

W2. The Stage 1–2 results support the benefit of LIFT-JEPA’s representation supervision before joint fine-tuning. However, its advantage over applying the same task emphasis directly to decoded predictions remains unclear, as Stages 2 and 3 use unweighted pixel MSE. A comparison with Vanilla JEPA using task-weighted observation-space losses on one representative radar dataset–backbone pair, under matched model-selection criteria and training budgets, would clarify the practical value of latent intervention weighting.

Questions for authors

Q1. On one radar dataset–backbone pair, how does LIFT-JEPA compare with Vanilla JEPA using task-weighted observation-space losses during decoding and fine-tuning, under the same validation criterion and training budget?

Q2. How sensitive are the weights and forecasting gains to the intervention type and strength? Could the responses reflect intervention artifacts rather than task relevance?

Q3. Do asa_s in Eq. (10) and aka_k in Fig. 1 denote the same scheduling coefficient? If so, please unify the notation.

Reviewer_6DD8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LIFT-JEPA, a task-aware objective for joint-embedding spatiotemporal forecasting. It constructs a task-intervened version of the factual future, encodes both versions with the same EMA target encoder, and uses their latent differences to weight the matching loss against the factual future representation. The method is evaluated on two radar nowcasting datasets and TaxiBJ with three forecasting backbones. Across all six radar dataset–backbone combinations, LIFT-JEPA improves mean CSI and CSI at the intervention threshold over Vanilla JEPA. The radar gains are accompanied by higher false-alarm rates.

Critical strengths
  1. The paper identifies a clear problem within JEPA training: task-critical regions can be specified in the observation space, but their importance cannot be directly assigned to individual spatial–channel latent units. The intervention-based weighting mechanism is intuitive and can be applied without changing the inference path. Retaining the full-field matching loss also ensures that supervision is not restricted to intervention-responsive units.
  2. Evaluation across three distinct backbones and two task definitions supports the method’s applicability beyond one architecture or one particular radar threshold. The uniform, random-mask, shuffled-weight, and mask-pooling controls provide useful evidence that the gains depend on the intervention and the assignment of weights to latent units.
Critical weaknesses
  1. LIFT-JEPA adds task emphasis during Stage 1, whereas Stages 2 and 3 train the decoder or full model with unweighted pixel MSE. Once decoded forecasts are available, the task mask can be applied directly to the pixel loss. The paper does not compare against task-weighted pixel training in Stage 3, so it is unclear whether latent intervention weighting provides benefits beyond this simpler approach. In addition, although the intervention branch adds no inference-time computation, Stage 1 requires an additional target-encoder pass over the intervened future. Reporting wall-clock training time, throughput, and peak GPU memory for Vanilla JEPA and LIFT-JEPA would help assess the practical value of the reported gains.

  2. If the encoder maps different inputs to the same representation and the predictor produces that representation, factual matching can be minimized while the intervention response, and hence the LIFT term, vanishes. This is a potential property of the stated objective, not evidence that collapse occurred in the reported experiments. The paper should explain which aspects of the architecture or training procedure prevent this behavior. Setting all above-threshold echoes to the no-echo value can create abrupt boundaries or holes within otherwise continuous structures. The resulting latent differences may reflect those new boundaries, the magnitude of the intensity change, or responses in neighboring no-echo regions, in addition to the removal of intense-echo information. Results with smoothly attenuated echoes or local interpolation, together with visualizations of response locations, would clarify what the weights capture.

Questions for authors
  1. What is the performance of the same backbone and three-stage protocol when the Stage-3 pixel MSE is weighted by the ground-truth intense-echo mask or TaxiBJ target-region mask? Does LIFT-JEPA still add a benefit when combined with that loss?
  2. What prevents the target encoder and predictor from converging to nearly input-independent representations under the normalized matching objective? Can the authors report representation variance or effective rank at the end of Stage 1?
  3. How sensitive are the results to the form of the radar intervention? In particular, do smoothly reducing intense echoes and replacing them by local interpolation produce similar latent weights and forecasting gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.