- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the proposed SphereJEPA learns to predict spherical video features from perspective inputs, providing a more comprehensive perception field. It uses spherical queries with spherical positional encoding to establish a persistent omnidirectional representation. SphereJEPA consists of a context encoder, an EMA target encoder, visibility-routed supervision, and a frozen V-JEPA teacher for correspondence supervision. The predictor is retained at inference to represent unobserved directions. Substantial experimental results evaluate the effectiveness of SphereJEPA on spherical binding, out-of-view feature consistency, scene retrieval, and downstream tracking tasks.
Critical strengths
-
A meaningful topic and a promising out-of-view representation-learning idea. Extending the spatial support of perspective video representations to a complete spherical omnidirectional space is an interesting and promising research. The combination of perspective-to-sphere binding and latent prediction provides a coherent way to represent both observed and unobserved directions without generating panoramic pixels. I find this representation level contribution worthwhile, even though the underlying architecture builds on V-JEPA.
-
Substantial and thoughtfully controlled experimental analysis. The authors conduct substantial experiments, which go beyond quantitative comparisons and qualitative feature visualizations. Visibility readouts, direction-centred retrieval, shuffled-scene controls, and Predict-Then-Observe provide meaningful evidence that the predicted features contain scene- and direction-dependent information. In particular, the 0.792 prediction–observation agreement, compared with 0.039 for shuffled predictions and 0.485 for raw sphere context, supports the usefulness of the predictor for this diagnostic. The 19.29° mean angular error on TAPVid360 also demonstrates promising coarse directional tracking without point-label training.
Critical weaknesses
-
The independent downstream contribution of out-of-view prediction remains unresolved. The experiments support the informative spherical features, but do not sufficiently establish whether learning to predict unobserved directions contributes to the reported tracking improvement. Table 4 varies correspondence supervision while retaining the out-of-view prediction objective. The Prediction–observation agreement remains relatively similar across variants, while tracking error increases substantially from 19.29° to 49.43°. This reflects the importance of correspondence supervision within the whole method, but does not disentangle its contribution from that of hidden feature prediction. A matched ablation removing the out-of-view prediction objective while retaining visible-region learning, visibility supervision, and the frozen correspondence teacher would directly address this gap. Since out-of-view prediction is central to the proposed method, this missing connection materially limits the experimental conclusions.
-
The paper’s presentation makes the contribution and evaluation pipeline difficult to assess. The introduction moves among maintaining out-of-frame information, predicting unobserved content, and establishing a spherical directional index without sufficiently distinguishing these objectives and connecting them to the proposed components. The method overview and Fig. 1 explain the EMA prediction framework, but the additional frozen correspondence teacher is introduced later without a sufficiently unified account of the complete training system. These organizational choices obscure the roles of binding, prediction, and correspondence supervision. The principal downstream readout also needs a more complete specification: which features are matched, how correspondences are selected, and how perspective-feature baselines are adapted to the Kabsch rotation-chain procedure. These details are important both for reproducibility and for determining whether tracking uses predicted hidden features directly or mainly benefits from improved visible correspondences.
Questions for authors
1.I am wondering what happens to downstream tracking when the out-of-view prediction loss is removed while keeping visible-region learning, visibility supervision, and the frozen correspondence teacher? Does the tracking readout directly use predictor-generated hidden features?
2.Please specify the features, matching/filtering rules, and ray correspondences used by the Kabsch readout for SphereJEPA and each frozen-feature baseline.
3.A same-scene, different-region evaluation would help to distinguish locally informative hidden-region predictions from direction-conditioned global scene information.
3 comments
I have carefully read the author's response, but based on comprehensive considerations, I still maintain my original attitude.
We thank the reviewer for reading our responses. We would be grateful to know which concerns remain, so that we can address them before the discussion period closes.
Reply to HuVc's review
We thank the reviewer for the thorough review, and are glad the reviewer found the topic meaningful and the experimental analysis substantial and carefully controlled.
Q1/W1. Does tracking use out-of-view prediction? Tracking reads the encoder's sphere tokens, because it needs precise matches between what the camera sees and its place on the sphere. The predictor has a different job, filling in the directions the camera has not seen, and its smoother features suit that job better than precise matching. We therefore test the two separately and have narrowed the claims to match. TAPVid360 tests how the encoder places what the camera sees on the sphere. The new semantic probe (general response, item 1) tests prediction directly, and there the predictor improves on the encoder's sphere tokens it starts from, from 43% to 49%. A matched model without Lpred would need retraining, which our compute budget does not allow.
Q2. Kabsch readout. For SphereJEPA, see our reply to h6VD (W3). The DINOv3 and V-JEPA 2.1 baselines have no sphere, so they treat frame 0 as the world frame, match each later frame's patches to it by nearest feature, and fit the rotation in the same way with the benchmark FoV. The paper now gives these definitions.
Q3. Same scene, different region. The new probe gives every baseline the clip's global features plus the query direction. SphereJEPA's per-direction predictions beat this by about 6 points, so they carry content specific to each region beyond knowing the scene.
W2. Presentation. We have clarified in the introduction how each property we evaluate maps to a component and a test. The encoder places what the camera sees on the sphere (TAPVid360), the predictor fills in unseen directions (the new semantic probe), and the index holds as the camera rotates (the temporal camera probe). The Fig. 1 caption now also describes the frozen correspondence teacher.
We believe these responses address the reviewer's remaining concerns, and we are happy to clarify anything further.