acceptodds
Withdrawn from ICLR 2027

PwP: Panoramas without Panoramas — Camera-Controlled 360° Video from Any View via Perspective-Only Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

360° video generation holds great promise for immersive applications such as virtual reality, yet precise control over camera motion in panoramic videos remains largely unexplored. Existing methods either lack explicit camera control or require a 360° panorama as the starting frame, which is costly to obtain and limits their practical applicability. In this work, we present a diffusion-based framework for camera-controllable 360° video generation that removes the dependence on panoramic start frames. Given a conditioning input and a target camera trajectory, our model generates temporally coherent 360° videos that faithfully follow the specified camera motion. To this end, we introduce a camera conditioning mechanism tailored to the spherical geometry of panoramic video, together with a training strategy that enables the model to synthesize full 360° content without panoramic input. Extensive experiments demonstrate that our method achieves state-of-the-art camera control accuracy while also improving visual quality and spatial consistency across viewpoints compared with existing approaches.

Primary area:
generative models
Keywords:
Diffusion Model, World Model, Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_ZtgzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a two-stage framework for camera-controlled 360-degree video generation from incomplete initial observations. Stage 1 learns a panoramic prior from complete general-domain ERP videos. Stage 2 adapts this model to driving using calibrated perspective multi-view videos projected into partial ERP targets. During evaluation, the model generates complete ERP videos from partial observation, a text prompt, and target trajectory. Experiments evaluate held-out self-driving sequences, cross-dataset transfer to nuScenes, and driving sequences from 360-1M.

Critical strengths
  • The adaptation mechanism addresses a useful practical setting: exploiting existing calibrated multi-view recordings without requiring complete target-domain panoramic ground truth. Masking unreliable supervision, including its spatial and temporal propagation into VAE latents, is a sensible way to avoid learning camera-boundary artifacts and treating missing regions as valid targets.

  • The paper is well-presented and easy to follow.

  • I love the website that contains abundant interactive visualizations .

Critical weaknesses
  1. The paper's claimed scope

The paper framed itself as general-propose panoramic video generation, but Stage 2 and the main adaptation experiments focus on driving videos. It is unclear whether driving is simply one example target domain for a general adaptation recipe, the intended scope of the contribution, or a source of training data expected to improve general panoramic generation. Besides, Stage I uses complete ERP videos, so the "panorama without panoramas" claim needs to remain explicitly limited to target-domain adaptation and inference requirements.

  1. Regarding Data

The reviewer does not fully agree with the authors' claim. The introduction reports approximately 150 suitable driving panoramas after filtering 360-1M, but does not disclose how this number is computed. As far as the reviewer know, Argus releases approximately 112k high-quality 10-second clip lists derived from 360-1M, while CubeComposer further curates this data source into 4K360Vid with 12k high-resolution clips and uses it for training. Although these are general-domain videos and do not directly contradict the "150 driving-only video" claim, the ratio between 150 and tens/hundreds thousand video is large, and the author should explicitly explain this.

Synthetic panoramas are another relevant alternative: game engines can render complete 360-degree sequences with known camera trajectories for driving and other scenes. Many existing papers (e.g., GenEx) used game engines such as Unreal Engine and Unity to synthesize high-quality 360 panoramic videos, either for self-driving or general propose domain. The paper neither adequately explain why such data are unsuitable, nor compared with baselines that make use of these synthesized data. A real-to-synthetic domain gap may matter, but it should be discussed and supported rather than assumed.

  1. Insufficient Experiments and Unclear Setting

Table 1 reports PwP (multiview) against baselines, while Figure 4 explicitly states that the baselines use single-view inputs and PwP uses multiview observations plus geometric control. This comparison mixes model differences with differences in visual coverage and conditioning. Section 5.1 claims a single-view PwP comparison, but Table 1 does not show the corresponding row. Moreover, "single-view" does not establish equivalent temporal information: these baselines are described as video-to-video methods, whereas PwP's single-view RGB condition is the first frame. The authors need to specify exactly which frames, views, camera information, and geometric inputs each method receives, and report a comparison with matched available information. The comparison also omits CubeComposer, a directly relevant recent perspective-to-360-degree video method with public inference code and checkpoints.

  1. Minor issues
  • Some bib entires contains page/volume information while others do not. It would be better to make the citation format consistent.
  • Figure 7 is not very easy to view in pdf.
Questions for authors
  1. Is driving intended as an example target domain or the principal application? Is the claim that Stage 2 improves driving generation while preserving the panoramic prior, or that it improves general panoramic generation? What evidence supports the latter, if intended?

  2. How was the approximately 150-video driving subset obtained? Please provide counts after each filtering step, distinguish source videos from clips, and explain its relationship to the public Argus/4K360Vid data. Why are existing driving-relevant panoramic clips and game-engine-rendered panoramic sequences insufficient or unsuitable alternatives?

  3. For every baseline and PwP variant in Table 1, what exact RGB frames and views are provided? Do the baselines receive a real perspective video or only an initial image? What data are used to reconstruct the point cloud and render the control sequence, and do they include future frames or additional views in the single-view setting? Please provide the missing single-view PwP results with matched conditioning.

  4. Can the authors add a comparison with CubeComposer or 360Anything (if they release code)?

Reviewer_9VBmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines scarce panoramic video datasets and abundant perspective video datasets to address panorama generation in driving scenarios. It adopts a two-stage training strategy: 1) learning panoramic priors from 360° data, and 2) adapting to partial perspective observations, allowing the model to synthesize panoramas from perspective views.

The method is reasonable and the video results are good. However, several concerns arise. The domain is too narrow: PwP is trained on numerous driving datasets, while the compared methods, e.g., Argus, ViewPoint, and Imagine360, are not specifically optimized on such data, so the generalization capability and evaluation results are not convincing. Besides, some of the technical details are missing, which confuses me.

Critical strengths
  1. The proposed method is reasonable: leveraging the strong spatial understanding capability from 3D models, lifting 2D videos into a 3D scene, and extracting 2D observations from the point cloud as a proxy, is a common solution in the field of controllable image/video generation.

  2. The video results seem good. I did not see noticeable seams or distortion in the panoramas.

Critical weaknesses
  1. The proposed method strongly relies on the capability of 3D models, i.e., PI3, in the paper. The authors make a hypothesis that PI3 is capable of reconstructing an accurate 3D point cloud given multi-perspective views. Though PI3 is trained in a permutation-equivariant manner, I doubt the reliability of the reconstruction, especially in stage 2, where complex motions (e.g., cars, passersby) and different recording conditions (e.g., FoV changes, color jumps — also mentioned in the paper) could cause severe degradation in PI3's performance.

  2. Section 3 is confusing. I didn't see the effect of the masking strategy. For example, even though a seam mask is applied, color jump still exists, and the authors did not test the effectiveness of such a design.

Questions for authors
  1. Are camera signals part of the DiT conditioning, or do they just externally control the observed point clouds that are sent to the DiT?
  2. How is the performance on OOD data?
  3. In Table 1, why is PwP evaluated on a multiview setting?
Reviewer_S8UUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates adapting camera-controlled 360° ERP video generators to driving domains where complete panoramic videos are unavailable. It proposes PwP, a two-stage framework that first learns a panoramic generation prior from curated 360-1M videos and then adapts it to inpaint partial ERPs to full ERPs from calibrated multiview driving videos. The adaptation employs reliability masks and masked latent-space rectified-flow supervision, together with point-cloud-based camera control. Experiments demonstrate improved generation quality over existing baselines on held-out driving sequences and cross-dataset evaluation on nuScenes, while largely preserving the panoramic generation capability learned in Stage I.

Critical strengths
  1. Practical problem and solution:* The paper identifies a gap between the data required for panoramic video generation and the data available in driving domains, where calibrated perspective multiview videos do not compose a complete ERP videos. Section 3 and Figure 2 explicitly account for incomplete observations and projection artifacts, rather than treating stitched multiview images as ground truth.

  2. Technically coherent adaptation mechanism supported by experimental evidence: The proposed masked latent-space supervision is well motivated. Equations (5)–(7) distinguish partial visual conditioning from valid latent regions used in the rectified-flow loss, while accounting for the spatial and temporal aggregation introduced by the video VAE. Figure 6 further demonstrates the applicability of the masking procedure across different camera configurations. Table 2 supports the effectiveness of the two-stage training strategy: PwP substantially outperforms Direct Stage II on both Test and nuScenes, while largely preserving the panoramic generation capability of Stage I on the held-out 360-1M evaluation.

Critical weaknesses
  1. Unfair baseline comparison: Although the paper claims that PwP outperforms existing methods under single-view conditioning, Table 1 only reports multiview results for PwP, while the baselines use single-view inputs. Moreover, PwP benefits from additional geometry-derived point-cloud control. These unequal conditions make it difficult to determine whether the improvements arise from the proposed adaptation strategy or richer input information.

  2. Insufficient validation of masked latent-space supervision: Equation (6) introduces spatial and temporal dilation to prevent invalid pixels from contaminating valid latent targets, but the paper does not explain how the dilation margins are selected or whether they adequately account for the video VAE's receptive field. Given that masked latent supervision is central to the method, further implementation details and ablations are needed to establish its reliability.

  3. Lack of consideration of ERP distortion: The method directly applies a conventional video diffusion architecture to ERP videos without explicitly addressing their inherent geometric distortion or periodic boundaries. The training objective also lacks latitude-aware weighting, potentially overemphasizing regions near the poles. The paper provides no analysis or ablation demonstrating whether the model can adequately handle these distortions, raising concerns about spatial consistency and generation quality across the full panorama.

Questions for authors

1.Single-view comparison: Section 5.1 claims that PwP outperforms existing methods under single-view conditioning, but Table 1 only reports multiview results. Could the authors provide the corresponding single-view results on Test and nuScenes? Furthermore, are the geometric controls available to the baselines under equivalent conditions? A comparison with matched inputs would help establish whether the improvements come from the proposed adaptation strategy rather than richer conditioning.

2.Masked latent supervision: Section 4.3 defines the VAE input as the masked observation Y~\tilde{Y}. How are invalid pixels represented in the VAE input, and how are the spatial and temporal dilation margins rs,rtr_s,r_t in Equation (6) selected? In particular, how do the authors ensure that valid latent targets are not contaminated by invalid regions through the VAE's receptive field? An ablation on the dilation margins would help validate this central design choice.

3.Evaluation protocol: For Table 1, could the authors clarify whether text prompts, sampling budgets, output resolutions, and camera trajectories are matched across PwP and the external baselines? Additionally, are the reported metrics computed over all generated frames, including the observed initial frame, or only future frames? These details are important for assessing the fairness of the comparison.

4.ERP distortion: Does the model explicitly account for geometric distortion and periodic boundaries in ERP representations, or does it rely entirely on implicit learning from Stage-I panoramic data? If no explicit mechanisms are used, could the authors provide evidence that the model maintains spatial consistency across different latitudes and the horizontal wraparound boundary? This would help assess the suitability of the underlying architecture for complete 360° video generation.

Reviewer_GPVEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PwP, a two-stage framework for generating camera-controlled 360 videos that adapts to a target domain (driving scenes in all experiments) without abundant driving-scene panorama data. In Stage 1, a video diffusion model is finetuned on curated ERP videos from the 360-1M dataset, utilizing a masked initial ERP frame and point cloud rendering. In Stage 2, by using multi-camera driving videos (Argoverse 2, Waymo Perception, Waymo E2E) as partial panorama videos and applying the rectified flow loss only to the valid locations, the VDM is further finetuned to generate driving 360 videos. The authors additionally claim a curation pipeline for web panoramas for 360-1M dataset. Evaluation uses 50-sequence held-out driving sets, nuScenes as an unseen dataset, and held-out 360-1M videos. The metrics for the driving sets are computed by projecting generated ERP frames into real camera views. PwP outperforms Argus, ViewPoint, and Imagine360 on the driving sets and GenEx, EvoWorld, and OmniRoam on 360-1M, and an ablation shows that Stage 1 initialization outperforms adapting the base Cosmos model directly with the masked Stage 2 loss.

Critical strengths

[S1] The paper is easy to read, and the overall idea and pipeline are straightforward to understand from the text and figures, although several details needed for reproduce are missing. The supplementary videos show visually convincing panoramic video quality. The motivation of the paper, exploiting abundant calibrated multi-camera data instead of scarce 360 videos is practically relevant.

Critical weaknesses

[W1] Limited novelty and an unverified adaptation mechanism.

Masked supervision of perspective views on the ERP canvas is already used for panoramic images in DiT360 (Feng et al., 2026), masked-loss fine-tuning of a video diffusion model on incomplete videos appears in ReCapture (Zhang et al., CVPR 2025)(not cited), and the point-cloud-rendering control follows GEN3C/ViewCrafter-style conditioning. The Stage 2 loss (Eq. 7) is the observed only objective that Ambient Diffusion (Daras et al., 2023) shows provides no learning signal in unobserved pixels. Their solution requires every pixel to be observed with non-zero probability (Daras et al., 2023, Thm. 4.1), which does not hold here. The poles lie outside every camera rig. PwP relies on the Stage 1 prior for these regions, and the Limitations paragraph (L484–486) acknowledges that the poles receive little supervision, but this is never verified in the target domain: driving metrics are computed only in the real camera views (L393–396), and the paper neither explains why the adaptation should work nor examines possible artifacts outside them. The method may work simply because the unsupervised regions are mostly sky and road in driving scenes, and the claim of a general target-domain adaptation framework is not supported. The contribution is a combination of existing components on top of a strong pretrained backbone (Cosmos-Transfer-2.5-2B), with little insight into why the masked adaptation works.

[W2] Limited evaluation and questionable comparisons.

(a) No camera-control evaluation. Despite the title, there is no camera accuracy metric (rotation/translation error from re-estimated poses), and all quantitative results use ground-truth trajectories. Novel trajectories appear only qualitatively (Fig. 1). Moreover, the OpenReview abstract differs severely from the PDF abstract and claims “state-of-the-art camera control accuracy,” which no experiment in the paper supports.

(b) Unclear inputs and potential information leakage. On the driving sets, the baselines are video-to-video methods (L424; Argus, Imagine360, and ViewPoint take a perspective video), yet L425–427 describes single-view PwP, which uses only the first frame of the center view, as "matching the input format of the perspective-conditioned baselines." Please clarify what each method receives. In either case, metrics are computed in the ground-truth camera views (L393–396), and PwP (multiview) observes all of them at t=0 while the baselines see only the front camera, so much of the gap in Table 1 (e.g., 19.67 vs. 11.61 PSNR) may reflect this input difference. The single-view PwP results that would make the comparison fair are claimed (L424–427; abstract, "even from a single view") but not reported. A second concern is the control signal. According to App. B.1 and Sec. 4.4 (L336–337), the point cloud is reconstructed from the initial views together with the 92 subsequent front-view frames of the same video. This matters most for 360-1M, where the baselines receive only the first panorama as visual input (L430–431). Since 360-1M has no camera annotations (L138–139), the "camera poses associated with each held-out sequence" (L373) presumably come from running this procedure on the test clip itself. If so, PwP's control signal contains geometry and appearance from future ground-truth frames that GenEx, EvoWorld, and OmniRoam never see.

(c) Missing driving-domain baseline. None of the external baselines is trained on the target driving datasets. Since target-domain metrics are computed only in the real camera views, a more appropriate baseline would be a driving multi-view video generator (e.g., MagicDrive-V2 or Cosmos multi-view) combined with panoramic inpainting of the uncovered regions. The paper itself positions ERP generation against such multi-view generators (L57–67) but never evaluates one in its multi-view setting.

Questions for authors

[Q1]. At test time, which frames are used to reconstruct the point cloud for the control sequence, for both the driving sets and 360-1M (App. B.1; Sec. 4.4, L336–337)? If future GT frames are included, please report results with a point cloud built only from the initial observation. Please also specify the exact inputs given to each baseline, and how the test-time camera poses for 360-1M are obtained. The answer to this question could change my rating.

[Q2]. Stage 1 data and architecture details are insufficient. How many curated 360-1M clips are used, and how many of the ~150 driving panoramas (L70) are included? How are the initial observation, mask, and control sequence injected into Cosmos-Transfer-2.5-2B, and how did the "Cosmos" baseline in Table 2 run on partial ERP inputs? Please also provide the key training details (e.g., steps, compute, rsr_s, rtr_t, and the mask pool) for reproducibility.

[Q3]. The data curation pipeline is claimed as a contribution, yet App. A.1 is a single paragraph listing six test names with no thresholds, retention statistics, or release plan. What exactly is the contribution here, and will the curated data or the foundation model be released?

[Q4]. Please report camera-control metrics on non-GT trajectories, together with 360°-specific quality measures (e.g., polar-region quality, metrics broken down by region).

[Q5]. How much does Stage 2 degrade the Stage 1 prior in regions it never supervises? Table 2 reports the driving sets only in the camera views, which Stage 2 supervises, and 360-1M only over the full ERP, so neither isolates the regions Stage 2 never supervises. Please report metrics restricted to regions outside a driving-rig mask, and/or a hold-out-camera evaluation in the driving domain (exclude one camera from both conditioning and the Stage 2 loss, then evaluate on that camera). Also, is the validity mask provided to the network, and how are the zero-filled regions of the noisy input (Eq. 5) handled during training?

The following are suggestions to the authors.

[Q6] Scope of claims. Unless results on another domain are provided, please scope the claims to driving-scene generation. Note that the introduction motivates full-ERP generation with "obstacles right under the ego car" (L81–82), yet these regions are never supervised in Stage 2, and are not evaluated.

[Q7] Two-stage design. Table 2 compares training with and without Stage 1, but not sequential versus joint training (e.g., mixing full-ERP 360-1M samples into Stage 2, as in DiT360's hybrid training). Please justify the sequential design or provide this ablation.

Reviewer_FPPAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PwP, a two-stage framework for camera-controlled 360° video generation without complete panoramic videos in the target domain. First, the model learns a panoramic prior from curated general-domain 360° videos. It is then adapted to driving scenes using partial equirectangular observations projected from calibrated multi-camera videos. The adaptation loss is restricted to reliable latent regions, and rendered point-cloud features provide camera-motion conditioning. The authors evaluate on held-out driving sequences, unseen nuScenes data, and held-out 360-1M panoramas, with comparisons against existing methods and an ablation of the two training stages.

Critical strengths
  1. The problem is practically important. Complete target-domain panoramic videos can be difficult to collect, whereas calibrated multi-camera data are available in domains such as autonomous driving. Adapting a panoramic generator using partial observations is therefore a useful direction.
  2. The two-stage approach is well motivated and supported by an initial ablation. Table 2 shows that combining panoramic initialization with target-domain adaptation improves over either Stage I or direct Stage II on the held-out driving and nuScenes evaluations.
Critical weaknesses
  1. The external comparisons do not adequately establish the claimed single-view advantage. Section 5.1 states that PwP outperforms the perspective-conditioned baselines under single-view conditioning, but Table 1 only reports the multi-view PwP results for the driving evaluations. Figure 4 also compares single-view baselines with multi-view PwP using geometry-derived control. Consequently, the reported comparison does not isolate the effect of the method from differences in available conditioning information. This matters because flexible operation from a single view is a central claim.
  2. The evidence for target-domain adaptation over the complete panorama is limited. On the driving datasets, reference-based metrics are computed on observed camera views, so they do not directly evaluate regions outside camera coverage. The complete-ERP evaluation on 360-1M is useful, but Table 1 reports PwP with a full-view input, which does not directly test completion of unseen regions from partial target-domain observations.
Questions for authors
  1. The authors may consider citing and discussing related works such as Image as a World (NeurIPS 2025), A Recipe for Generating 3D Worlds From a Single Image (2025), and WonderWorld (CVPR 2025).
  2. Stage II uses multi-camera driving datasets with a limited number of fixed camera layouts, whose coverage is concentrated around the horizontal viewing band. Although the nuScenes experiment provides evidence of cross-dataset generalization, it remains unclear how well the model generalizes to substantially different camera configurations or observation masks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.