acceptodds
Under review as a conference paper at ICLR 2027

SphereJEPA: Learning Spherical Video Representations From Perspective Video via Self-Supervision

Abstract

We introduce SphereJEPA, a joint-embedding predictive architecture that learns to predict features for every direction around the camera from ordinary perspective video. Its context and target come from different projections of the same scene. The context encoder reads the perspective video, an EMA target encoder reads the panorama, and both write onto tokens indexed by direction on a gravity-aligned sphere. Unlike a standard JEPA, whose predictor is discarded after pre-training, SphereJEPA keeps its predictor, because its output is the representation of what lies out of view. Training crops are rendered from the panorama, so the data needs no posed or calibrated capture, and the model never receives camera parameters. Inference is a single forward pass through a 113M-parameter encoder and predictor. The predicted features identify unseen scenes and their orientation, and agree with what the camera later observes. With frozen features and an analytical readout, SphereJEPA roughly halves the TAPVid360 mean angular error of every method not trained on the benchmark, without using point labels. representation whose spatial support extends beyond the current image, while fine-grained transfer depends on the complete training construction.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, joint-embedding predictive architecture, video representation learning, 360° video, spherical representations, out-of-view prediction, point tracking

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_HuVcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the proposed SphereJEPA learns to predict spherical video features from perspective inputs, providing a more comprehensive perception field. It uses spherical queries with spherical positional encoding to establish a persistent omnidirectional representation. SphereJEPA consists of a context encoder, an EMA target encoder, visibility-routed supervision, and a frozen V-JEPA teacher for correspondence supervision. The predictor is retained at inference to represent unobserved directions. Substantial experimental results evaluate the effectiveness of SphereJEPA on spherical binding, out-of-view feature consistency, scene retrieval, and downstream tracking tasks.

Critical strengths
  1. A meaningful topic and a promising out-of-view representation-learning idea. Extending the spatial support of perspective video representations to a complete spherical omnidirectional space is an interesting and promising research. The combination of perspective-to-sphere binding and latent prediction provides a coherent way to represent both observed and unobserved directions without generating panoramic pixels. I find this representation level contribution worthwhile, even though the underlying architecture builds on V-JEPA.

  2. Substantial and thoughtfully controlled experimental analysis. The authors conduct substantial experiments, which go beyond quantitative comparisons and qualitative feature visualizations. Visibility readouts, direction-centred retrieval, shuffled-scene controls, and Predict-Then-Observe provide meaningful evidence that the predicted features contain scene- and direction-dependent information. In particular, the 0.792 prediction–observation agreement, compared with 0.039 for shuffled predictions and 0.485 for raw sphere context, supports the usefulness of the predictor for this diagnostic. The 19.29° mean angular error on TAPVid360 also demonstrates promising coarse directional tracking without point-label training.

Critical weaknesses
  1. The independent downstream contribution of out-of-view prediction remains unresolved. The experiments support the informative spherical features, but do not sufficiently establish whether learning to predict unobserved directions contributes to the reported tracking improvement. Table 4 varies correspondence supervision while retaining the out-of-view prediction objective. The Prediction–observation agreement remains relatively similar across variants, while tracking error increases substantially from 19.29° to 49.43°. This reflects the importance of correspondence supervision within the whole method, but does not disentangle its contribution from that of hidden feature prediction. A matched ablation removing the out-of-view prediction objective while retaining visible-region learning, visibility supervision, and the frozen correspondence teacher would directly address this gap. Since out-of-view prediction is central to the proposed method, this missing connection materially limits the experimental conclusions.

  2. The paper’s presentation makes the contribution and evaluation pipeline difficult to assess. The introduction moves among maintaining out-of-frame information, predicting unobserved content, and establishing a spherical directional index without sufficiently distinguishing these objectives and connecting them to the proposed components. The method overview and Fig. 1 explain the EMA prediction framework, but the additional frozen correspondence teacher is introduced later without a sufficiently unified account of the complete training system. These organizational choices obscure the roles of binding, prediction, and correspondence supervision. The principal downstream readout also needs a more complete specification: which features are matched, how correspondences are selected, and how perspective-feature baselines are adapted to the Kabsch rotation-chain procedure. These details are important both for reproducibility and for determining whether tracking uses predicted hidden features directly or mainly benefits from improved visible correspondences.

Questions for authors

1.I am wondering what happens to downstream tracking when the out-of-view prediction loss is removed while keeping visible-region learning, visibility supervision, and the frozen correspondence teacher? Does the tracking readout directly use predictor-generated hidden features?

2.Please specify the features, matching/filtering rules, and ray correspondences used by the Kabsch readout for SphereJEPA and each frozen-feature baseline.

3.A same-scene, different-region evaluation would help to distinguish locally informative hidden-region predictions from direction-conditioned global scene information.

3 comments
AuthorsOpenReview ↗

Reply to HuVc's review

We thank the reviewer for the thorough review, and are glad the reviewer found the topic meaningful and the experimental analysis substantial and carefully controlled.

Q1/W1. Does tracking use out-of-view prediction? Tracking reads the encoder's sphere tokens, because it needs precise matches between what the camera sees and its place on the sphere. The predictor has a different job, filling in the directions the camera has not seen, and its smoother features suit that job better than precise matching. We therefore test the two separately and have narrowed the claims to match. TAPVid360 tests how the encoder places what the camera sees on the sphere. The new semantic probe (general response, item 1) tests prediction directly, and there the predictor improves on the encoder's sphere tokens it starts from, from 43% to 49%. A matched model without Lpred\mathcal L_{\mathrm{pred}} would need retraining, which our compute budget does not allow.

Q2. Kabsch readout. For SphereJEPA, see our reply to h6VD (W3). The DINOv3 and V-JEPA 2.1 baselines have no sphere, so they treat frame 0 as the world frame, match each later frame's patches to it by nearest feature, and fit the rotation in the same way with the benchmark FoV. The paper now gives these definitions.

Q3. Same scene, different region. The new probe gives every baseline the clip's global features plus the query direction. SphereJEPA's per-direction predictions beat this by about 6 points, so they carry content specific to each region beyond knowing the scene.

W2. Presentation. We have clarified in the introduction how each property we evaluate maps to a component and a test. The encoder places what the camera sees on the sphere (TAPVid360), the predictor fills in unseen directions (the new semantic probe), and the index holds as the camera rotates (the temporal camera probe). The Fig. 1 caption now also describes the frozen correspondence teacher.

We believe these responses address the reviewer's remaining concerns, and we are happy to clarify anything further.

I have carefully read the author's response, but based on comprehensive considerations, I still maintain my original attitude.

We thank the reviewer for reading our responses. We would be grateful to know which concerns remain, so that we can address them before the discussion period closes.

Reviewer_XKnFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SphereJEPA, a joint-embedding predictive architecture that maps ordinary perspective video to a feature field over all directions around the camera, indexed on a gravity-aligned sphere locked to the first frame. A context encoder, initialized from V-JEPA 2.1 ViT-B, jointly attends over perspective patches and learnable sphere query tokens, so it must infer where observed content belongs without camera parameters. An EMA target encoder reads the full equirectangular video and produces targets on the same spherical grid. A visibility head decides which directions count as observed, and a predictor completes the rest. Unlike in a standard JEPA, the predictor is kept at inference. Training inputs are perspective crops rendered from 360° video, so camera parameters are used only to build supervision. The model is evaluated on held-out 360° clips in three ways: an analytical visibility readout for crop placement, same-model retrieval and a predict-then-observe protocol for out-of-view content, and a parameter-free Kabsch readout on TAPVid360. On TAPVid360 it reports 19.29° mean angular error, compared with 37.43° for CoTracker3 and 43.54° for frozen DINOv3 features.

Critical strengths

The paper extends the spatial support of a perspective-video representation from the current image plane to the full directional field around the camera. It also turns 360° video into a free source of supervision: perspective inputs are rendered, so no posed or calibrated capture is needed. At inference the model needs no camera parameters and produces a latent representation of the whole sphere in a single forward pass, avoiding the cost of generating pixels. The paper also report some rigorous diagnostics and also report unfavourable results honestly.

Critical weaknesses
  • The paper's central claim lacks downstream validation: the central claim is that the predictor's output is a representation of out-of-view content, yet no downstream task actually uses these out-of-view features:

  • TAPVid360 effectively measures camera rotation estimation. TAPVid360 is the only external benchmark. Its readout estimates camera rotation from matches in the visible region only, then propagates query directions by that rotation under a static-point assumption. It therefore mainly tests rotation estimation and the stability of crop-to-sphere binding, not out-of-view prediction. Even so, its threshold accuracy (0.0442) is below DINOv3 with the same Kabsch readout (0.0632) and far below dedicated trackers (around 0.24). The abstract's claim of roughly halving the error holds only for mean angular error.

  • The out-of-view evaluation is purely intrinsic. It consists only of same-model retrieval and cosine agreement, always comparing against the model's own features. It is also small in scale: 188 clips, of which 147 are eligible for predict-then-observe, with only 11 clips in the 90°–180° bin beyond the visible boundary.

  • There is no semantic or geometric probing. The paper does not probe out-of-view features semantically or geometrically, for example by training a linear head on panoramic depth or segmentation data and scoring visible and out-of-view regions separately.

  • There is no comparison with generative methods. The paper motivates itself against perspective-to-panorama generators such as Argus and 360Anything, arguing that they are expensive and do not capture uncertainty. Yet it never compares against them, either through a generate-then-encode baseline using the same target encoder or through a runtime comparison. Moreover, SphereJEPA regresses a single target with an L1 loss, so it also yields one deterministic completion, and the uncertainty argument does not favour it.

  • There are no embodied or closed-loop tasks, even though the motivation repeatedly relies on them.

Questions for authors
  • Usefulness of out-of-view features. Can a linear head trained on out-of-view predicted features recover panoramic semantic or depth labels (for example on Stanford2D3D, Matterport3D, or Pano3D)? Could the author compare with baselines that first generates a panorama with Argus or 360Anything and then perform each corresponding task (with the V-JEPA 2.1 or other methods)?

  • Preservation of representation quality. Under V-JEPA 2.1's standard protocols (attentive probes on SSv2 and K400, linear probes on ADE20K and NYUv2, label propagation on DAVIS), how does the SphereJEPA encoder perform relative to its V-JEPA 2.1 ViT-B initialization?

  • Architectural choice. Did you try encoding first and then decoding with sphere queries? With the encoder sequence growing from 4,608 to 17,152 tokens, how much do inference latency and memory increase relative to V-JEPA 2.1 ViT-B?

  • Generalization to real perspective video. All video evaluations, including TAPVid360, use perspective views rendered from 360° video, while transfer to real single photographs degrades substantially. How well does crop-to-sphere binding hold on perspective video captured by real cameras with ground-truth orientation? Real cameras introduce lens distortion, different aspect ratios, and a different imaging pipeline. This bears directly on whether the "from perspective video" claim in the title holds.

2 comments
AuthorsOpenReview ↗

Reply to XKnF's review

We thank the reviewer for the detailed review, and are glad the reviewer valued the camera-free, single-pass design and our reporting of unfavourable results. We ran the suggested generate-then-encode baseline (W4).

W1. TAPVid360 measures rotation. We agree, and the abstract now says that TAPVid360 tests placing the view on the sphere and that the halving refers to mean error. Its thresholds are all finer than one of our sphere tokens (6.4°), so we added coarser ones.

% of points within5°10°20°20° (out-of-frame points only)
DINOv3, nearest feature20354415
SphereJEPA15346553

SphereJEPA matches DINOv3 at 10° and keeps the direction far more often beyond that, especially out of frame, while DINOv3 is more precise below one token.

W2/W3/Q1. Extrinsic validation. See general response, item 1. It uses an independent segmenter's labels and scores only never-seen directions.

W4. Generative baseline. We ran Argus with ground-truth cameras, its most favourable setting, and encoded its panorama with our own target encoder. SphereJEPA's predictions are 7 points better (general response, item 1). Argus matches us next to the visible region and falls behind further away, because one sampled completion is often wrong far from the frame. It also takes about 400× longer. We agree on uncertainty, since our L1 target also gives one completion, and have removed that claim.

Q2. Representation quality. We ran DAVIS label propagation. SphereJEPA scores 61 J&F against 67 for its V-JEPA 2.1 initialisation, so it keeps most of its initialisation's correspondence quality while adding the spherical index. The paper now reports this.

Q3. Architecture and cost. See tgpE (W5) for cost. We chose one joint sequence because, without camera parameters, a separate decoder would have to learn the whole image-to-sphere mapping in cross-attention, while the joint sequence lets every layer work on it.

Q4. Real perspective video. We agree that every video test renders views from 360° video, and that the photo benchmarks show a transfer gap (we found its main cause, see tgpE, W4). The abstract and limitations now state this.

W5. Embodied tasks. We present embodied use as a motivating application, and the paper's claims concern the representation itself. The introduction now makes this clearer, and we list closed-loop evaluation as future work.

We believe these responses address the reviewer's remaining concerns, and we are happy to clarify anything further.

We thank the authors for the detailed response. The new segmentation probe and the Argus generate-then-encode baseline partly address our main concern about out-of-view validation, and we appreciate the clarified scope of the TAPVid360 claim. However, several important issues remain. Most importantly, the modified encoder shows a clear loss of representation quality: DAVIS drops from 67 to 61 J&F relative to the V-JEPA 2.1 initialization, a significant degradation for a representation learning paper. The other standard benchmarks we requested (attentive probes on SSv2 and K400, linear probes on ADE20K and NYUv2) are still not reported, so the overall cost of the modification remains unclear, on top of a roughly 7× increase in inference cost. The new probe is also limited in scope. It uses pseudo-labels on only 188 clips and reports accuracy alone, which is dominated by layout priors (direction alone reaches 32%). It gives no confidence intervals, includes no row that isolates the predictor's contribution, and leaves the probe's training protocol unspecified. In addition, the revised description of the preprocessing implies that the "gravity-aligned" sphere is in fact only aligned to the 360° camera's own orientation, and the residual tilt is not quantified. Finally, all video evaluations still use views rendered from 360° video, so generalization to real perspective video is untested. We therefore keep our current score.

Reviewer_h6VDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper adopts a JEPA self-supervised training framework that samples perspective (pinhole) video clips from 360° panoramic videos. Given only a sampled pinhole video, the network is trained to reconstruct the complete 360° spherical feature field and predict which region of the panorama the pinhole view corresponds to. This equips pinhole video representations with broader 360° spatial awareness and allows the model to predict full panoramic features from ordinary pinhole inputs to support several downstream tasks: (1) estimating camera FoV and 3D orientation on the gravity-aligned sphere; (2) out-of-view pinhole-to-panorama scene and azimuth retrieval; and (3) label-free 3D directional tracking on TAPVid360. Unlike standard 2D point trackers that lose track once an object leaves the frame, SphereJEPA matches camera-frame pinhole features against canonical sphere features to estimate 3D camera rotation, continuously tracking the target's 3D unit direction dt∈R3d_t \in \mathbb{R}^3 in the current camera frame even out of view. Overall, these tasks clearly demonstrate the advantages and potential of this 360°-aware self-supervised representation

Critical strengths

Learning a 360° surrounding representation from a narrow-FoV pinhole video is very well-motivated and closely mirrors human spatial perception: although we only see a limited forward field of view at any moment, we naturally maintain an awareness of the environment around and behind us, and the proposed network demonstrates this capability well.

The self-supervised JEPA design is also cleverly combined with panoramic cameras. Whereas traditional JEPAs mask patches within a pinhole view to predict missing latent features in that same planar view, SphereJEPA exploits the natural correspondence between perspective crops and panoramas by sampling pinhole video clips from a 360° video to predict the complete 360° spherical feature field. Because no camera parameters are given, solving this task forces the model to jointly infer both where the pinhole view sits on the sphere and what scene features lie beyond its borders;

Finally, the experiments on TAPVid360 3D directional point tracking—where the model continues to estimate a target point's 3D direction even after it completely leaves the frame—clearly highlight the advantages of panoramic features. And because this label-free setup requires no calibrated camera rigs and can construct abundant training pairs by sampling diverse virtual pinhole trajectories from uncalibrated web 360° videos, this representation shows strong potential for scaling up.

Critical weaknesses

1. Mismatched positional-encoding units between pinhole and panorama tokens, and no ablation on SPE(θ, φ).

In Encoder-x (Section 3.1, Eqs. 1–2), pinhole tokens use 2D (row, col) RoPE in pixel-grid coordinates, while panorama tokens do not use spatial RoPE and instead add spherical harmonics SPE(θ, φ) in angular coordinates. When pinhole and panorama tokens attend to each other in shared self-attention, taking dot products between 2D pixel-grid RoPE and 3D spherical harmonics has no clear mathematical meaning because they are not even in the same unit. In addition, the paper has no ablation study on the new panorama positional encoding SPE(θ, φ) (Table 4 only ablates Lcorr\mathcal{L}_{\text{corr}}).

What happens if panorama tokens simply use the same (t, row, col) RoPE (or standard 2D positional embeddings) as pinhole tokens? Please provide an ablation study to show the actual benefit of SPE(θ, φ).

2. Unclear inter-frame rotation estimation on 360° videos and missing failure analysis.

Both training and the 188-clip validation set assume that all frames in a 360° video have zero relative rotation (locked to frame 0) and are gravity-aligned with GeoCalib (Section 3.1, Section S1). However, the paper does not explain how inter-frame rotations are estimated accurately across 10-second 360° videos, or how GeoCalib (which is built for pinhole images) is run on panoramas. In real web videos with moving objects, fast motion blur, or textureless skies, pose estimation can easily fail, yet Section S1 only mentions removing missing or undecodable video files.

How do you obtain accurate inter-frame rotations and gravity alignment on 360° videos? Are there failure cases, how often do they happen, and how do you filter them out?

3. Unspecified source of pinhole camera intrinsics in TAPVid360 Kabsch tracking.

In Section 4.3 (Table 3, p. 8) and Section S3 (p. 17), TAPVid360 tracking matches 2D pinhole patches to 3D sphere tokens and uses Kabsch alignment to solve for 3D camera rotation. However, converting 2D pinhole patches into 3D rays for Kabsch alignment requires the pinhole camera intrinsics (Field of View, FoV). The paper does not state where these camera intrinsics come from.

When running Kabsch alignment on TAPVid360, do you use the ground-truth camera intrinsics from the TAPVid360 dataset, or intrinsics predicted by SphereJEPA? Please provide the exact equations and report the result using model-predicted FoV.

Questions for authors

4. Pre-training uses 25% TAPVid-style camera trajectories that match the test distribution.

Section S2 (p. 16) shows that 25% of pre-training camera trajectories are generated using "TAPVid-style motion". Since this motion distribution is very similar to the TAPVid360 test set, it is unclear how much of the 19.29° tracking score comes from matching this synthetic motion prior.

How much does TAPVid360 performance drop if you remove the 25% TAPVid-style trajectories from pre-training (for example, training only on real camera motions from DynPose-100K)?

5. Generalization to wide-angle or fisheye camera videos.

Full 360° videos are relatively scarce, often contain stitching artifacts, and require 12,544 sphere tokens (~6.43° per token) to cover the whole sphere. In contrast, wide-angle and fisheye videos (120°–170° FoV, e.g., driving, or egocentric datasets) are much larger in scale and have better image quality.

Can the same training method be applied to wide-angle or fisheye videos (cropping narrow pinhole clips to predict features on a partial spherical region), or does the framework strictly require full 360° panoramas?

6. Blind FoV estimation easily overfits; supporting known camera intrinsics would unify positional encodings and improve robustness.

Estimating FoV from a monocular narrow-FoV video alone is inherently difficult and easily overfits to the training data. Supplementary Table S4 (p. 22) clearly shows this problem: FoV error is 3.84° on in-distribution validation clips (Table S3), but jumps to 18.23° on NYUv2, 10.27° on ETH3D, and 28.25° on iBims-1. In real applications, camera intrinsics are actually easy to obtain (from calibration or EXIF metadata). If the model took camera intrinsics as input, each pinhole patch could be converted into a 3D camera ray. Then both pinhole and panorama tokens could use spherical harmonics in the same angular unit—instead of mixing (u, v) pixel-grid RoPE with spherical harmonics, where the network does not even know the relative scale between the pinhole view and the panorama. This should also make cross-dataset generalization much more robust.

Why not build a version of SphereJEPA that accepts camera intrinsics as input, using 3D camera rays to unify the positional encodings of pinhole and panorama tokens into spherical harmonics?

7. Missing mathematical definitions and implementation details.

Several details needed for full clarity and reproducibility are omitted from the main text and appendix: (a) the explicit equation for the correspondence loss Lcorr\mathcal{L}_{\text{corr}} in Section 3.2 (p. 5); (b) the exact formula for the visibility routing weight rkr_k after step 10k and at inference (Section 3.1 p. 4 implies a hard {0, 1} switch, whereas Section 3.2 p. 5 and Eqs. S4–S6 describe soft "blending"); and (c) how the 45% "simulated or interest-guided motion" trajectories are generated and what temporal stride / FPS is used to sample 16 frames from 10s clips (Section S2)

8. Same-source GeoCalib pseudo-label bias in video-level evaluation.

During pre-training, all 360° videos are gravity-aligned using GeoCalib pseudo-labels, so the network naturally fits GeoCalib's output distribution and any systematic biases it may carry. However, in the 188-clip video evaluation (Section 4.1 and Table S3), the reference "ground-truth" labels are also produced by GeoCalib in the exact same way (while external ground-truth datasets in Appendix Table S4 only test duplicated single images rather than real videos). Using GeoCalib pseudo-labels as test-time ground truth may reward fitting GeoCalib's bias rather than true physical gravity; could the authors discuss this evaluation limitation, and clarify why video-level gravity/pose evaluation was not checked against independent physical measurements (such as IMU, LiDAR, or SfM sensors) or predictions from a different gravity estimator?

Overall, I believe the points above—especially the missing mathematical details and design clarifications—are straightforward to fix, and the authors do not need to run additional experiments. If the authors can provide clear and reasonable explanations in the rebuttal, I will be happy to raise my score.

2 comments
AuthorsOpenReview ↗

Reply to h6VD's review

We thank the reviewer for the careful reading and the precise questions, and are glad the reviewer found the problem well motivated and the label-free setup promising for scaling. We have added each missing definition to the paper.

W1. Positional encodings. Attention never compares a RoPE angle with a spherical harmonic. Sphere tokens get the same temporal RoPE as perspective tokens and no spatial rotation, and their direction enters only through SPE added at the input. A perspective-to-sphere score is therefore an ordinary dot product between a position-rotated query and a key that encodes its direction. The network learns which directions each image position should attend to, which it has to learn anyway because the camera is unknown. We chose spherical harmonics because ERP row and column coordinates break at the ±180° seam and distort near the poles. An ablation would need retraining, so we list it as a limitation.

W2. Rotation and gravity. Rotations are estimated, not assumed. We run COLMAP and GLOMAP on cubemap faces of every second frame and rotate each frame to match the first. We removed clips where stabilisation had failed (0.5% of training clips), identified by an earlier model's camera error. GeoCalib was run on eight horizontal views and their roll and pitch averaged, which leaves the sphere aligned to each clip's first frame. It is therefore upright as far as the source camera was level, which we measured at a median of about 3°. The paper now describes the frame this way.

W3. TAPVid360 intrinsics. We used the benchmark's FoV, as did both Kabsch baselines. For each tubelet, every perspective patch jj is matched to its most similar sphere token, and the camera rotation is the best fit between the patch rays and the matched directions, Rt=arg⁡min⁡R∈SO(3)∑j∥dk∗(j)−Rcj(f)∥2,k∗(j)=arg⁡max⁡kcos⁡(ht,jp,ht,ks),R_t=\arg\min_{R\in SO(3)}\sum_j\big\Vert\mathbf d_{k^{\ast}(j)}-R\mathbf c_j(f)\big\Vert^2,\quad k^{\ast}(j)=\arg\max_k\cos(\mathbf h^p_{t,j},\mathbf h^s_{t,k}), with no filtering. Rotations are interpolated to all frames and each query direction is carried by them. With SphereJEPA's own FoV estimate the error is 19.5° against 19.3°. FoV matters little because the rotation depends mainly on where the image centre lands on the sphere.

Q4. TAPVid-style motion. No TAPVid360 data was used. This family is hand-written parametric motion (spins, sweeps, spirals) chosen to resemble the benchmark, so it is a matched prior and the paper now says so. Adding it to an earlier model improved TAPVid360 by about 3.5°, which is small next to our 24° lead over the baselines.

Q5. Fisheye video. The method should apply if the prediction loss is restricted to directions the source camera covers, so fisheye video would supervise a hemisphere.

Q6. Known intrinsics. We chose camera-free inference because much web and robot video is uncalibrated. A version that takes camera rays as input is a natural extension, and we now mention it. During the rebuttal we found the main cause of the photo transfer gap (see tgpE, W4), and it is one that known intrinsics would avoid.

Q7. Definitions.

  • (a) Lcorr=1−∑kwkcos⁡(hkctx, ∑pWkptp)∑kwk\mathcal L_{\mathrm{corr}}=1-\frac{\sum_k w_k\cos(\mathbf h^{\mathrm{ctx}}_k,\ \sum_p W_{kp}\mathbf t_p)}{\sum_k w_k}, where tp\mathbf t_p is the frozen V-JEPA 2.1 feature of perspective patch pp, WkpW_{kp} is the fraction of sphere token kk that projects into patch pp, and wkw_k weights visible tokens by their area on the sphere.
  • (b) Routing uses the ground-truth mask until step 5k, switches gradually to the predicted mask by step 10k, and from then on (and at inference) blends by the predicted visibility, rk=sg(pkvis)r_k=\mathrm{sg}(p^{\mathrm{vis}}_k). We corrected the wording that implied a hard switch.
  • (c) We sample 16 frames at 4 fps (3.75 s) from a random start. The trajectory generators and their parameters are now in Appendix S2.

Q8. GeoCalib in evaluation. The video camera metrics use no GeoCalib labels. Their ground truth is the rendered camera, measured in the same frame the model was trained in. The single-image benchmarks give the independent check, since their gravity comes from SfM or calibrated capture. There SphereJEPA's pitch error is far below an upright guess (8.7° against 22.8° on MegaDepth), so its sense of up matches real gravity. Video with IMU ground truth would be a stronger test, which we list as future work.

We believe these responses address the reviewer's remaining concerns, and we are happy to clarify anything further.

Reviewer_h6VDOpenReview ↗

Response to Authors' Rebuttal

Thank you for the clear responses. The added equations and implementation details address my concerns well. I really like the core idea of combining panoramas with JEPA, as the formulation is very clever and the motivation is strong. While I am not yet fully sure whether it warrants a 4, I am happy to raise my score to 3: Weak acceptance, as promised.

Reviewer_LSN3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SphereJEPA, a JEPA-based framework that learns spherical feature representations from perspective RGB videos. The method uses spherical query tokens to represent directions around the camera and predicts features for regions outside the observed field of view. It builds on V-JEPA 2.1 and introduces spherical positional encoding, visibility-based routing, and correspondence supervision to align perspective and spherical representations.

The framework is evaluated on scene and direction retrieval, predict-then-observe feature agreement, and directional tracking using TAPVid360. The authors also provide ablation studies investigating the contribution of correspondence supervision.

Critical strengths
  • The paper addresses an interesting problem of predicting representations for unseen regions from limited visual observations. This could be useful for scene understanding and embodied AI.

  • The proposed framework combines JEPA-based representation learning with spherical representations, visibility prediction, and geometric correspondence supervision. This provides an interesting approach to learning directional features without requiring camera parameters at inference.

  • The experimental results are promising, particularly for out-of-view feature prediction and directional tracking. The predict-then-observe evaluation provides useful evidence that the predicted features contain scene-specific information.

  • The paper provides useful ablation studies on correspondence supervision. The results show substantial improvements in tracking performance when using correspondence loss and a frozen pretrained teacher.

Critical weaknesses
  1. Limited analysis of the contribution of the proposed spherical architecture

The paper introduces spherical query tokens, spherical positional encoding, and visibility-routed prediction on top of the V-JEPA 2.1 framework. However, it remains unclear how much of the overall performance improvement comes from these architectural components rather than the additional geometric supervision. The ablation studies in Section 4.4 provide useful evidence for the contributions of correspondence supervision and the frozen pretrained teacher. In particular, the comparison between no correspondence, online correspondence, and the full construction helps distinguish these effects. However, all three variants share the same spherical architecture. Therefore, the individual contribution of the spherical representation itself is not fully established. A comparison with a simpler V-JEPA-based approach using comparable geometric supervision would help clarify the benefit of the proposed architecture

  1. Limited direct comparison for out-of-view feature prediction

The main objective of SphereJEPA is to predict spherical features, including regions outside the camera's field of view. The experiments provide promising results using scene retrieval, direction retrieval, and predict-then-observe agreement. However, comparisons with existing methods are mainly presented through the downstream TAPVid360 benchmark. While these results demonstrate the usefulness of the learned representations, it remains unclear how much SphereJEPA improves out-of-view feature prediction compared with simpler JEPA-based approaches or other existing methods.

  1. Interpretation of the directional tracking results

The TAPVid360 results show a substantial reduction in mean angular error compared with generic pretrained feature baselines. However, the threshold accuracy remains relatively low and is comparable to those baselines. This suggests that SphereJEPA may be more effective at estimating approximate directions than providing precise point localisation. Although the authors acknowledge this limitation, I think the practical implications of this difference could be discussed more clearly.

Questions for authors
  1. Contribution of the spherical architecture: Section 4.4 provides useful ablations distinguishing the contributions of correspondence supervision and the frozen pretrained teacher. However, all variants use the same spherical architecture. Could the authors clarify how much of the improvement comes from the spherical representation itself? Would a simpler V-JEPA-based model trained with comparable geometric supervision achieve similar performance?

  2. Out-of-view prediction comparison: Since the main objective is to predict features outside the camera's field of view, could the authors clarify how SphereJEPA compares with simpler JEPA-based methods or other relevant baselines on this specific task, rather than primarily through downstream TAPVid360 tracking?

  3. Tracking precision: The TAPVid360 results show a substantial improvement in mean angular error, but threshold accuracy remains relatively low. Could the authors clarify what level of directional localisation the model can reliably achieve and what types of downstream applications would benefit from this level of precision?

  4. Use of the predictor: The paper states that standard JEPA methods discard the predictor after pretraining. However, some JEPA-based methods retain predictors for downstream prediction. Could the authors clarify this statement and explain what specifically distinguishes the use of the predictor in SphereJEPA from previous approaches?

1 comment
AuthorsOpenReview ↗

Reply to LSN3's review

We thank the reviewer for the positive assessment, and are glad the reviewer found the problem interesting and the predict-then-observe evaluation and ablations useful.

W1/Q1. Contribution of the spherical architecture. The new probe gives the closest comparison. V-JEPA 2.1 features given the query direction reach 43% on never-seen directions, against 49% for SphereJEPA. A matched V-JEPA model with geometric supervision would need retraining beyond our compute budget.

W2/Q2. Out-of-view comparison. See general response, item 1. SphereJEPA's predictions beat both V-JEPA 2.1 and a generate-then-encode baseline using Argus.

W3/Q3. Precision. TAPVid360's thresholds are finer than one sphere token. Within 20°, SphereJEPA tracks 65% of points against 44% for the best baseline, and 53% of out-of-frame points. That suits deciding where to look or re-finding an object that left the view, and it complements a point tracker, which can refine the direction once the object is back in view.

Q4. The predictor. I-JEPA and V-JEPA discard the predictor. World-model JEPAs keep one to roll the state forward in time. SphereJEPA keeps its predictor to fill in unseen directions at the same instant, and the introduction now makes this distinction.

We believe these responses address the reviewer's remaining concerns, and we are happy to clarify anything further.

Reviewer_tgpEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Overall, the paper uses panoramic video for supervised learning of spherical features outside the field of view, achieving good direction tracking results without using point tracking annotations for training. The idea is clear and it has certain practical value. However, how much benefit the out-of-field-of-view prediction brings, whether the prediction content is reliable, require more comprehensive experiments to verify. The cross-dataset performance, implementation details, and actual costs also need to be further explained.

Critical strengths

The author used panoramic video to supervise the spherical feature prediction of perspective video. During the inference process, the predictor was retained without inputting camera parameters. Overall, the paper's thinking was relatively clear and straightforward. Moreover, after removing the direction mean, the accuracy rates of scene and orientation detection reached 85.9% and 88.5% respectively. The method did not use point tracking annotations for training. The average angular error of TAPVid360 was 19.29°, and the effect was also good.

Critical weaknesses
  1. In Table 4 of the paper, the training settings were controlled, and the corresponding supervision effect was also verified. However, at the same time, the additional contribution and effect of monitoring outside the field of view on tracking were not isolated. While retaining the teacher and visible area supervision, removing the prediction loss outside the field of view, and comparing the tracking results with and without the completion feature.
  2. In Section 4.2 of the paper, the detection and the observation after prediction are all evaluated within the model's own feature space. Although it is helpful after removing the mean and random order control, the verification of hidden content is still not direct enough. An independent feature extractor or the evaluation of the hidden area semantic task needs to be added.
  3. In Section 4.3 of the paper combined with Appendix S3, the source of the field of view angle of TAPVid360 for matching spherical features and the screening rules for corresponding relationships were not clearly explained. Further explanation and disclosure are needed for these details. These details will affect the results in Table 3. Therefore, the author needs to provide a complete process content.
  4. In Table S4 of the paper, the field of view angle error of the continuous probe on NYUv2 was 18.23°. This is higher than 360Anything's 3.90°. Although you discussed the differences in the appendix, I suggest fixing the backbone and retraining the probe using the single image for matching to distinguish the impact of representing ability and inability to read mismatch.
  5. The number of parameters and the word prefix cannot indicate the actual cost. It is recommended that the author supplement the peak memory usage, GPU utilization, and efficiency comparison under the same conditions, etc.
Questions for authors

Look at the weaknesses section

1 comment
AuthorsOpenReview ↗

Reply to tgpE's review

We thank the reviewer for the constructive review, and are glad they found the idea clear and of practical value. The single-image probe the reviewer suggested led us to the main cause of the photo transfer gap (W4).

W1. Isolating out-of-view prediction. Tracking reads the encoder's sphere tokens, while the predictor fills in the directions the camera has not seen, so we test the two separately (see HuVc, Q1). W2 tests prediction directly.

W2. Independent check of hidden content. We did this (general response, item 1). Labels come from an ADE20K segmenter that shares nothing with our model, and we score only directions the camera never saw. SphereJEPA's predictions reach 49%, against 43% for V-JEPA 2.1 and 54% for an encoder that sees the hidden region.

W3. TAPVid360 FoV and matching. See our reply to h6VD (W3) for the equations. With SphereJEPA's own FoV the result is unchanged.

W4. Single-image probe. We refitted the probe on single frames with the backbone frozen. It transferred to photographs no better, which pointed to our training data instead. Our narrow-FoV training crops are upsampled from 1024-pixel-wide video and so are blurry, and the model learnt to read FoV from sharpness. Sharp photographs therefore look wide. As a diagnostic, blurring each photograph to match a training crop of its FoV cuts the NYUv2 error from 26° to 3°. Pitch, which this does not affect, already transfers. Rendering training crops at higher resolution would remove this cue.

W5. Cost. Measured on one RTX 4090 for a 16-frame clip:

ModelLatencyPeak memory
V-JEPA 2.1 ViT-B21 ms1.4 GiB
SphereJEPA157 ms2.3 GiB
Argus (generation only)60 s12.5 GiB

SphereJEPA predicts the full sphere in 157 ms, several hundred times faster than generating the panorama. Its extra cost over V-JEPA 2.1 comes from attending over the dense sphere grid.

We believe these responses address the reviewer's remaining concerns, and we are happy to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading and constructive comments. We are glad that tgpE found the idea clear and of practical value, noting good direction tracking without point-tracking labels, and that LSN3 found the out-of-view results promising and the ablations in Section 4.4 useful. We also appreciate h6VD's view that the remaining issues are straightforward to fix. Every new result below uses the submitted checkpoint. We summarise the main changes here, respond to each reviewer's points in a separate reply, and welcome further discussion of the paper.

1. A direct test of out-of-view prediction. Several reviewers asked whether the predicted features really contain the hidden part of the scene, judged by something other than our own model. We labelled the full panoramas of our 188 held-out clips with an off-the-shelf ADE20K segmenter. A small probe then reads the label at directions the camera never saw. Every row gets the same probe and the query direction, and only the features change.

Features given to the probeAccuracy on never-seen directions (%)
Direction only32
V-JEPA 2.1, whole clip43
Argus panorama (given true cameras), encoded by our target encoder42
Argus panorama, segmented directly43
SphereJEPA prediction49
Our target encoder on the real panorama (upper bound)54

SphereJEPA's predictions beat V-JEPA 2.1 and the Argus generate-then-encode baseline, and close most of the gap to the encoder that sees the hidden region. Argus needs about a minute per clip, while SphereJEPA needs 0.16 s.

2. Clearer description of the implementation. While answering the reviews we revised several descriptions so they match the implementation exactly. None changes a reported number.

  • The TAPVid360 tracker matches the encoder's sphere tokens, without using the predictor's output. TAPVid360 therefore tests how well the model places what it sees on the sphere, and item 1 is our test of prediction. The abstract now says this.
  • Tracking used the benchmark's FoV. Using SphereJEPA's own FoV estimate gives the same error (19.5° against 19.3°).
  • Our GeoCalib step averaged roll and pitch over views spaced around the panorama, which cancels a tilted camera. The sphere is therefore aligned to each clip's first frame and is only as upright as the 360° camera was, typically within a few degrees. We now describe it that way.
  • We excluded 0.5% of training clips whose stabilisation had failed, which the paper did not mention.

3. Other new results, each detailed in our reply to the reviewer who asked.

  • Coarser TAPVid360 thresholds (XKnF, W1). SphereJEPA keeps 65% of points within 20°, against 44% for the best baseline.
  • DAVIS label propagation (XKnF, Q2). SphereJEPA scores 61 J&F against 67 for its V-JEPA 2.1 initialisation.
  • Inference cost (tgpE, W5). SphereJEPA takes 157 ms per clip, about 7× V-JEPA 2.1 and several hundred times less than Argus.
  • Single-image camera probe (tgpE, W4). The transfer gap to photographs comes mainly from blurry narrow-FoV crops in our training data.
  • TAPVid360 readout equations (h6VD, W3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.