acceptodds
Under review as a conference paper at ICLR 2027

PVWander: 3D Memory-guided Panoramic Video Wandering with Path-Facing Canonicalization

Abstract

Long-horizon panoramic video wandering aims to continuously synthesize scene-consistent observations along user-specified camera trajectories from an initial equirectangular projection (ERP) panorama. Existing panoramic video models have achieved promising results in camera-controlled generation, while recent 3D-guided approaches further provide geometric guidance for improving scene consistency. However, stable long-horizon wandering remains challenging due to diverse motion–orientation relationships and imperfect memory guidance induced by accumulated generation artifacts and depth inaccuracies. To address these challenges, we propose **PVWander**, a 3D-memory-guided framework that accumulates generated observations into an online point-cloud memory and reprojects it to guide subsequent generation, with two complementary designs: *Path-Facing canonicalization* for motion–orientation alignment and *Timestep-aware gating modulation* for adaptive memory guidance. *Path-Facing canonicalization* exploits the horizontal cyclic symmetry of ERP panoramas to uniformly align each clip's ERP orientation with its principal translation direction, yielding more consistent, forward-like relative motions while preserving the prescribed camera path. *Timestep-aware gating modulation* exploits the coarse-to-fine nature of diffusion denoising to adaptively regulate reprojected memory guidance through timestep-dependent channel-wise gates, preserving its structural benefits while mitigating visual artifacts from imperfect projections. Extensive experiments demonstrate that PVWander achieves improved long-horizon generation quality, while ablations validate the effectiveness of both designs. Code will be available.

Primary area:
generative models
Keywords:
Panoramic Video Generation, Long-Horizon Video Generation, Camera-Controlled Video Generation, 3D Memory, Video Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ktFvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PVWander, a framework that first lifts 2D panoramic videos to 3D point clouds, and re-renders videos from partially observed point clouds. Path-Facing Canonicalization aims to regularize the ERP direction, and Timestep-aware Gating is proposed to eliminate artifacts. The idea is intuitive but not novel, and the writing is easy to follow. There is no supplementary material, and no video results are provided.

Critical strengths
  1. The proposed design is reasonable, and the writing is easy to follow.
  2. Quantitative experiments demonstrate the effectiveness of PVWander.
Critical weaknesses
  1. The Path-Facing Canonicalization seems somewhat trivial, as it mainly operates on the yaw axis, though the authors provide an ablation study on this design. This is equivalent to horizontally rotating an ERP, while the main visual content still lies within the ERP grid. However, when it comes to pitch/roll movement, trivial visual content (e.g., the ceiling) takes over the ERP grid, and severe distortion appears. How robust is PVWander in this situation?

  2. No video results are provided, making it difficult to evaluate the quality of the generated panoramic videos, which is a critical aspect in this field.

Questions for authors

How robust is PVWander? Does PVMetric fail when sharp pitch or roll is applied?

5 comments

We sincerely thank Reviewer ktFv for the valuable feedback. We have prepared an anonymized project page with additional qualitative results and videos: https://review-supplement.pages.dev. Below, we address each weakness (W) and question (Q) point by point. We will incorporate the relevant content into the revised manuscript.

W1 & Q1: Robustness to Large Pitch/Roll Rotations

1. Handling large pitch/roll rotations

In our task setting, target camera poses are precisely known. We therefore perform generation and depth estimation with zero pitch&roll, then use spherical reprojection (e.g., Equilib [1]) to obtain the requested viewing orientations. Unlike yaw, which can be handled by ERP's horizontal cyclic shift as in Path-Facing, pitch/roll substantially alter ERP distortion and may introduce a distribution gap for pretrained models. This setting is also consistent with recent methods such as OmniRoam[2] and PanoWorld[3].

Accordingly, below we evaluate the visual robustness of reoriented videos and depth estimation under large pitch/roll rotations.

2. Robustness evaluation of PVWander and PVMetric

We evaluate both the visual quality of reoriented panoramic videos (Tab. I) and the accuracy of depth estimation (Tab. II) under large pitch/roll rotations.

For PVWander, we conduct experiments on 98 InteriorGS-OmniRoam test scenes, each containing 641 frames. Pitch and roll continuously vary over time within [−60°,60°][-60°,60°] throughout each sequence. Identical spherical rotations are applied to the generated panoramas and corresponding GT, using the same rotation trajectories for both PVWander and OmniRoam.

Table I. Robustness of output reorientation on 98 test scenes (8 segments per scene).

MethodPitch/Roll rangeSSIM ↑PSNR ↑LPIPS ↓
PVWander0°0.681016.320.3983
PVWander±60°0.670416.030.4071
OmniRoam0°0.606314.590.4663
OmniRoam±60°0.593214.360.4741

As shown in Tab. I, large pitch/roll reorientations introduce only minor degradation, while PVWander maintains better generation quality than OmniRoam under both upright and rotated settings.

For depth estimation, PVMetric is specifically trained for our generation framework using panoramas with zero pitch&roll. We evaluate PVMetric under large pitch/roll rotations and compare it with the general-purpose panoramic depth estimator DAP [4].

Table II. Depth estimation accuracy under pitch/roll rotations (±60°). For upright inference + output rotation, we estimate depth from upright input and then apply spherical rotations to the predicted depth maps.

SettingPVMetric Abs Rel ↓PVMetric δ₁ ↑DAP Abs Rel ↓DAP δ₁ ↑
Upright input (0°)0.06240.95410.15160.8462
Rotated input0.24270.53000.30190.4187
Upright inference + output rotation0.06080.95700.14740.8488

Directly feeding rotated ERP panoramas to depth estimators leads to substantial accuracy degradation. DAP exhibits a similar trend, indicating that sensitivity to large pitch/roll variations is a broader challenge in panoramic depth estimation. Notably, performing depth estimation on upright panoramas followed by spherical reorientation nearly restores the original accuracy for both estimators.

Section 3 of our anonymous project page further provides videos, demonstrating the reorientation of PVWander outputs to different pitch and roll angles.

[1] PyEquilib: Processing Equirectangular Images with Python, https://github.com/haruishi43/equilib, 2021

[2] Omniroam: World wandering via long-horizon panoramic video generation. SIGGRAPH 2026

[3] PanoWorld: Real-World Panoramic Generation. ArXiv 2026

[4] Depth any panoramas: A foundation model for panoramic depth estimation. CVPR 2026

W2: Video supplementary

We have created an anonymized project page: https://review-supplement.pages.dev. Please refer to Section 2 for video results.

AuthorsOpenReview ↗

Response to the Novelty Concern (Summary & W1)

Response to the Novelty Concern (Summary & W1)

Our core contributions target two challenges: diverse motion–orientation relationships and imperfect 3d memory guidance.

For the former, exploiting the horizontal cyclic symmetry of ERP, we propose Path-Facing to align each segment's ERP orientation with its principal motion direction. The key lies in the trajectory-dependent canonicalization, rather than the cyclic shift operation itself. For the latter, we propose Timestep-aware Gating to adaptively regulate memory guidance across denoising stages, preserving useful structural benefits while mitigating unreliable local details.

In our manuscript, the statistics over 31K sampled clips in Fig. 1 show that Path-Facing effectively reduces the variability of motion directions relative to ERP orientations. Fig. 1 and Fig. 6 further illustrate the timestep-dependent behavior of the learned gating output, consistent with our coarse-to-fine motivation. Tab. 3 demonstrates the contributions of both designs through ablations, while Tab. 4 further supports the gating design details.

Importantly, Path-Facing is not tied to the 3D memory pipeline or a specific framework. Tab. 5 demonstrates its zero-shot transfer to existing panoramic generators, including Wan2.1-Fun. Furthermore, Tab. 3 shows that incorporating Path-Facing during training brings additional improvements to Wan2.1-Fun.

Overall, these results support the practical value and transferability of our designs.

2. Additional comparison with EvoWorld.

To further evaluate the generalizability of our designs, we conduct additional experiments on the Unity_curve dataset of EvoWorld, as shown in Tab. III.

First, to control for the backbone difference, we implement an SVD-based PVWander variant and compare it with the official EvoWorld. At the matched 1024×512 resolution, SVD-based PVWander consistently outperforms the official EvoWorld across all metrics at both 25 and 73 frames.

Furthermore, to test whether our designs can benefit an independent framework, we reproduce EvoWorld and integrate Path-Facing (PF) and Timestep-aware Gating (TAG) into it. Under identical training and inference settings at 640×320, adding PF and TAG improves all metrics over the reproduced EvoWorld baseline. After high-resolution fine-tuning at 1024×512, the enhanced model also outperforms the official EvoWorld.

Overall, these results demonstrate that our designs generalize across different backbones and can effectively benefit an independent panoramic framework. We will include these comparisons in the revised manuscript.

Table III. Comparison on EvoWorld's outdoor Unity_curve dataset. 25F and 73F denote the first-segment (25 frames) and three-segment (73 frames) evaluations, respectively. PF: Path-Facing Canonicalization; TAG: Timestep-Aware Gating. All reproduced models follow the same training schedule: 5K steps (batch size 24) at 640×320, followed by 3K fine-tuning steps at 1024×512. Other settings follow the official EvoWorld implementation.

MethodResolution25F-SSIM↑25F-PSNR↑25F-LPIPS↓25F-FID↓25F-FID_cube↓25F-TransErr↓25F-RotErr↓73F-SSIM↑73F-PSNR↑73F-LPIPS↓73F-FID↓73F-FID_cube↓73F-TransErr↓73F-RotErr↓
EvoWorld (official, native)1024×5760.774518.930.323814.6519.700.76810.100.748016.880.419225.5323.683.46743.70
EvoWorld (official, resized)1024×5120.772319.060.321614.5419.700.76810.100.744216.960.418125.3023.683.46743.70
PVWander (SVD-based)1024×5120.819922.000.199112.0119.520.0690.2620.789120.040.259713.0217.421.79518.15
EvoWorld* (reproduced)640×3200.732119.260.303420.6827.510.5758.2060.690516.870.420536.0934.883.54045.12
EvoWorld* + PF & TAG640×3200.774521.190.215614.1120.770.2390.3380.741719.380.271516.9718.811.83618.27
EvoWorld* + PF & TAG1024×5120.809121.130.226511.3817.890.2330.2740.783419.430.270511.5015.061.82718.32

Thank you for your response. Considering that the novelty does not convince me, I have decided to maintain my score.

Reviewer_mPndOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PVWander, a framework for long-horizon panoramic video generation along user-specified camera trajectories. PVWander generates video segment by segment and maintains an online 3D colored point-cloud memory that accumulates generated observations and reprojects them to guide subsequent generation. The paper introduces two main components: (1) Path-Facing Canonicalization, which reorients equirectangular projection (ERP) panoramas to align with the principal translation direction (obtained via PCA); and (2) Timestep-aware Gating Modulation, an MLP that learns timestep-dependent channel-wise gates for memory features, enabling stronger memory guidance in early diffusion stages and weaker reliance during late refinement. The authors also introduce PVMetric, a feed-forward panoramic video metric-depth estimator.

Experiments on InteriorGS-OmniRoam and zero-shot loop-closure tests on RealSee3D show improved long-horizon consistency and appearance quality over Matrix-3D, OmniRoam, and PanoWorld.

Critical strengths

The two main contributions of the paper (Path-Facing canonicalization and Timestep-aware gating) are well-motivated, and the experiments and ablations provide good insight into their impact.

The comparisons are well designed and convincing. For example, the authors retrain the Matrix-3D baseline on the same backbone under two depth protocols (MoGe and PVMetric) (Tabs. 1 and 6), which separates the contribution of persistent memory from differences in backbone, data, and depth source. They also apply Path-Facing at inference to the baselines (Tab. 5), complementing the ablation in Tab. 3.

Critical weaknesses
  • Related work is not discussed in depth in the main paper.

  • The gains from individual components are not always consistent. In Tab. 3, adding Timestep-aware Gating or Path-Facing to online memory slightly lowers SSIM.

  • The memory accumulates observations continuously, yet the paper neither discusses the impact of memory growth nor analyzes computational complexity.

Questions for authors
  • Could the authors provide an ablation comparing PVMetric, MoGe, and GT depth within the full PVWander framework?

  • How are degenerate segments, such as rotation-only motion, near-zero displacement, or U-turns, handled? How does Path-Facing behave on such trajectories, and does RealSee3D include such cases?

  • How does performance scale beyond 8 segments, e.g., to 16 or 32?

  • How would the method perform in dynamic scenes or under changing illumination?

5 comments

We sincerely thank the Reviewer mPnd for the thoughtful and constructive feedback. We have prepared an anonymized project page with additional qualitative results: https://review-supplement.pages.dev. Below, we address each weakness (W) and question (Q) point by point. We will incorporate the relevant content into the revised manuscript.

W1: Related work

Thank you for this suggestion. Due to the original page limit, we placed some of the more detailed discussions of related work in the Appendix to prioritize the presentation of our method and experiments. With an additional page available for the revision, we will move the relevant discussions back into the revised manuscript.

W2: Gains from individual components

Compared with online memory alone, adding Gating or Path-Facing individually decreases SSIM by 0.0036 and 0.0039, respectively, while both consistently improve LPIPS (0.4132 → 0.4042 / 0.4107), FID (26.69 → 23.79 / 23.09), and FID_cube (42.67 → 40.06 / 40.50).

This trade-off may be because the generated details, although perceptually clearer, do not necessarily align spatially with the ground truth. SSIM measures local luminance, contrast, and structural similarity, making it sensitive to local correspondence; sharper or more natural-looking results therefore do not always yield higher SSIM.

To illustrate this discrepancy, we provide qualitative comparisons in Section 1 of Anonymous Project Page. As shown in Fig. 1 on the project page, the online-memory baseline exhibits substantial smoothing and detail degradation, yet achieves higher SSIM than the Gating output (0.6244 vs. 0.6010), despite worse LPIPS (0.6180 vs. 0.4577). Furthermore, as shown in Fig. 2, applying Gaussian blur to the same Gating output frame increases SSIM to 0.6332 while worsening LPIPS to 0.5781. Similar discrepancies between structural similarity metrics and perceptual quality have also been reported in prior studies [1, 2].

Therefore, generated video quality should be assessed by considering complementary metrics that capture different aspects of structural fidelity and perceptual quality.

Importantly, Path-Facing and Gating exhibit complementary effects when combined: the full model improves all metrics over online memory alone, including SSIM. This demonstrates that their combination achieves a better balance between structural similarity and perceptual quality.

[1] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR 2018

[2] Image Quality Assessment: Unifying Structure and Texture Similarity. TPAMI 2022

W3: Memory Growth and Computational Complexity

Thank you for raising this point. Historical memory stores NN RGB observations of PP pixels each, requiring O(NP)O(NP) storage space. Each new segment selects at most K=4K=4 observations by target-view coverage. Depth estimation and source selection scale with NN, while reprojection costs O(TKP)O(TKP) for a segment of TT frames.

1. Stage-wise computational cost We profile the memory-conditioning pipeline on an A100 40GB GPU at 720×1440 (Tab. I).

Table I. Stage-wise runtime of the memory-conditioning pipeline.

Target SegmentHistorical FramesDepth Estimation (s)Source Selection (s)Reprojection (s)VAE Encode/Decode (s)Denoising (s)
Segment 2817.530.120.7747.35472.80
Segment 424121.330.310.8347.35472.80
Segment 856150.170.690.7847.35472.80

Depth estimation accounts for the main increase in cost because we reprocess all historical RGB observations with PVMetric at each rollout stage to improve depth consistency.

2. Overall computational complexity.

We further compare different methods for an 8-segment rollout at 720×1440 resolution on an A100 40GB GPU (Tab. II).

Table II. Computational cost comparison across different methods.

MethodBackboneTFLOPs / 8 SegmentsRuntime (min / 8 Segments)
Wan2.1-FunWan2.1-1.3B646k66.84
PVWanderWan2.1-1.3B657k72.76
OmniRoam(Preview + Refine)Wan2.1-1.3B5231k488.63
Matrix3DWan2.2-5B400k48.83
PanoWorldWan2.2-5B217k41.54

Interestingly, although Matrix3D and PanoWorld use larger 5B backbones, their computational costs and runtimes are lower. This is mainly because Wan2.2-5B employs a VAE with a higher spatial compression ratio, resulting in fewer latent tokens. OmniRoam, in contrast, concatenates Preview conditioning latents with noisy video latents, doubling the token count and substantially increasing attention computation.

For an eight-segment PVWander rollout, diffusion denoising, VAE encoding/decoding, and accumulated depth processing take approximately 63.04, 6.31, and 3.40 minutes, respectively. Thus, although memory-related costs grow with history length, they do not constitute the primary computational bottleneck in the current 8-segment setting.

Q1: Different depth source ablation for PVWander

We conduct an additional depth-source ablation within the full PVWander framework. All configurations include Path-Facing and Timestep-aware Gating, with 1–4 source frames randomly sampled during training. Please note that since panoramic depth estimation with MoGe is computationally expensive (about 2min per panorama on an A100), it is impractical to use it for large-scale training-time depth estimation as we do with PVMetric.

Therefore, in addition to our original PVMetric-trained/PVMetric-inference configuration, we train an additional model using GT depth and evaluate it with GT, MoGe, and PVMetric depth at inference time.

As shown in Tab. III, GT/GT serves as an oracle reference and achieves the best overall performance. Among configurations using predicted depth at inference, PVMetric/PVMetric achieves the best performance. This indicates that adapting the generator to depth estimation errors during training helps it better utilize geometric guidance at inference. This finding is consistent with our observations in original manuscript.

Table III. Depth-source ablation for PVWander. All results are evaluated on the first 81-frame segment, as GT depth is unavailable for subsequent segments. “w/ align” denotes aligning the predicted depth to the GT depth scale.

Train DepthInference DepthSSIMPSNRLPIPSFIDFID_Cube
GTGT0.741621.840.150617.1031.89
GTMoGe (w/ align)0.678819.850.196718.4033.94
GTPVMetric (w/ align)0.687420.120.188618.5034.36
PVMetricPVMetric0.735321.440.161517.9233.08

Q2: Degenerate segments

Handling different motion patterns. Path-Facing determines a shared reference orientation using PCA on current entire trajectory. When no reliable principal direction can be identified, it falls back to the first frame's orientation. For rotation-only motion, since the camera center remains fixed, target views can be directly obtained through spherical reprojection of the existing panorama, without requiring video generation.

Evaluation on specifically designed trajectories. Although RealSee3D includes revisitation trajectories, its discrete observations lack frame-wise ground truth along complete trajectories, preventing quantitative evaluation of intermediate results. We therefore use the 3DGS representations of 98 test scenes from InteriorGS-OmniRoam to render ground truth along specifically designed trajectories. We construct 240 trajectories, including 80 U-turns, 80 closed-return trajectories with near-zero net displacement, and 80 zig-zag trajectories, each containing 81 frames.

To isolate the effect of Path-Facing, we compare three configurations: (A) the original Wan2.1-Fun baseline; (B) Path-Facing applied only at inference time (zero-shot); and (C) Path-Facing applied during both training and inference.

Table IV. Evaluation of Path-Facing under challenge trajectories.

ConfigurationSSIM ↑PSNR ↑LPIPS ↓FID ↓
A: Baseline0.772523.840.08769.58
B: + PF, zero-shot0.773623.880.08729.21
C: + PF, train & infer0.776023.950.08759.03

As shown in Tab. IV, both Path-Facing configurations maintain comparable or slightly better generation performance than the baseline across all four metrics, despite frequent changes in local motion directions and ambiguous dominant translation directions.

Q3: Performance scale beyond 8 segments

Following the camera trajectories provided by InteriorGS-OmniRoam, we select 20 test scenes that support longer trajectory rendering and extend the original 8 segments by rendering 8 additional segments, constructing evaluation sequences of 16 segments in total.

Tab. V compares the generation performance between the first 8 segments and the subsequent 8 segments. As expected, all methods exhibit error accumulation and quality degradation as the rollout length increases. Nevertheless, PVWander consistently outperforms the baselines, demonstrating its long-horizon generation advantage beyond the original evaluation length.

Please note that we do not evaluate 32 segments, as evaluating all methods on 32 segments for 20 scenes would require approximately 1000 A100 GPU hours, making it infeasible within the rebuttal period.

Table V. Long-horizon generation performance over 16 segments. Results are reported separately for the first 8 segments and the subsequent 8 segments, evaluated on the same 20 scenes.

MethodResolutionSeg. 1–8 SSIM↑Seg. 1–8 PSNR↑Seg. 1–8 LPIPS↓Seg. 9–16 SSIM↑Seg. 9–16 PSNR↑Seg. 9–16 LPIPS↓
PVWander1440×7200.718617.540.34160.687513.540.5576
Matrix-3D1440×7200.600013.500.54340.50099.030.7466
OmniRoam1440×7200.625214.880.45010.582213.030.5593
PVWander960×4800.682917.590.35610.647413.590.6008
PanoWorld960×4800.469212.400.57760.33389.530.7368
OmniRoam-P960×4800.574414.250.45580.44509.490.6667

Q4: Dynamic Scenes and Changing Illumination

PVWander is trained exclusively on static scenes and has no specific mechanism for handling dynamic objects. Consequently, moving objects may be accumulated as static content in the 3D memory and reprojected in subsequent segments. Regarding illumination, our datasets do not cover varying lighting conditions, and the framework has no explicit illumination conditioning. Extending panoramic world generation to dynamic environments with appropriate memory updating mechanisms, as well as incorporating explicit illumination conditioning for controllable lighting variations, are important directions for future research.

Reviewer_da4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes pvwander, a framework for long-horizon panoramic video wandering given a single panorama and camera trajectory. Built on wan2.1-fun-1.3b-control-camera, it keeps an online 3D point-cloud memory from generated frames and reprojects it as render condition for the next segment. On top of this, two designs are proposed: path-facing canonicalization, which rotates the panorama so each segment moves along its main translation direction, and timestep-aware gating, which modulates the memory render tokens with timestep dependent channel-wise gates. Experiments on interiorgs-omniroam and zero-shot realsee3d show improvements over matrix-3d, panoworld and omniroam.

Critical strengths
  • The paper addresses an interesting problem. Long-horizon panoramic wandering with consistent scene memory is useful for world models and VR.
  • Path-Facing is simple, parameter-free, and can be directly applied to other panoramic generators without retraining.
  • The paper is easy to follow
Critical weaknesses
  • Novelty is limited. Most of the memory pipeline comes from existing works: online point-cloud memory like spatial memory [1] and evoworld [2], coverage based source selection following lyra 2.0 (also similar to anchorweave [3]), additive render tokens following matrix-3d, and ERP plucker rays following campvg. The two proposed designs are relatively small tricks, a yaw circular shift based on the PCA direction and a timestep MLP gate on the render tokens. Overall it feels more like an engineering combination than a new idea. The closest work, evoworld [2], also maintains an evolving 3D point-cloud memory for panoramic generation. It is discussed in the appendix but not compared in experiments.

  • The gains of the two designs are small and not consistent. In Tab. 3, adding gating to online memory actually lowers SSIM, and Path-Facing + online memory is also lower than online memory alone. In Tab. 5, Path-Facing only improves SSIM by 0.003~0.01, and some metrics even get worse. Without multiple runs or variance it is hard to tell whether these differences are significant. The memory is still a globally fused point cloud, so depth errors and generation artifacts keep accumulating, and the gating only softly downweights them (also shown in the failure case in Fig. 7). Also Path-Facing relies on a clear principal translation direction, its unclear how it works for rotation-heavy, zig-zag or back-and-forth trajectories, which are common in free wandering.

  • The evaluation is limited. Training and main evaluation are on synthetic indoor scenes (InteriorGS) only. Matrix-3d and panoworld are single-segment models extended by last-frame rollout, which are relatively weak baselines for long-horizon memory. The zero-shot test only has 20 scenes, and the loop metric only compares the first and the last frame.

  • There is no project page or video supplementary, and the paper only shows one or two qualitative examples (Fig. 4 and Fig. 5). For a long-horizon video generation paper, it is very hard to judge temporal consistency and revisit quality from a few sampled frames

[1] Video World Models with Long-term Spatial Memory. NeurIPS 2025. [2] EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory. arXiv 2025. [3] AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories. ECCV 2026.

Questions for authors

See weaknesses.

7 comments

We sincerely thank Reviewer da4g for the valuable feedback and suggestions. We have prepared an anonymized project page with additional qualitative results: https://review-supplement.pages.dev. Below, we address each weakness (W) point by point. We will incorporate the relevant content into the revised manuscript.

W1: Novelty Concern

1. For memory pipeline

The 3D memory pipeline serves as the foundation of our framework rather than a standalone contribution. All prior works that informed this pipeline are cited in the manuscript. We will further cite AnchorWeave in the revised manuscript.

2. For two proposed designs

Our two core designs target two challenges: diverse motion–orientation relationships and imperfect 3d memory guidance. For the former, exploiting the horizontal cyclic symmetry of ERP, we propose Path-Facing to align each segment's ERP orientation with its principal motion direction. For the latter, we propose Timestep-aware Gating to adaptively regulate memory guidance across denoising stages, preserving useful structural benefits while mitigating unreliable local details.

In our manuscript, the statistics over 31K sampled clips in Fig. 1 show that Path-Facing effectively reduces the variability of motion directions relative to ERP orientations. Fig. 1 and Fig. 6 further illustrate the timestep-dependent behavior of the learned gating output, consistent with our coarse-to-fine motivation. Tab. 3 demonstrates the contributions of both designs through ablations, while Tab. 4 further supports the gating design details.

Importantly, Path-Facing is not tied to the 3D memory pipeline or a specific framework. Tab. 5 demonstrates its zero-shot transfer to existing panoramic generators, including Wan2.1-Fun. Furthermore, Tab. 3 shows that incorporating Path-Facing during training brings additional improvements to Wan2.1-Fun.

Overall, these results support the practical value and transferability of our designs.

3. Comparison with EvoWorld

To further evaluate the generalizability of our designs and address the missing comparison with EvoWorld, we conduct additional experiments on its outdoor Unity_curve dataset, as shown in Tab. I.

First, to control for the backbone difference, we implement an SVD-based PVWander variant and compare it with the official EvoWorld. At the matched 1024×512 resolution, SVD-based PVWander consistently outperforms the official EvoWorld across all metrics at both 25 and 73 frames.

Furthermore, to test whether our designs can benefit an independent framework, we reproduce EvoWorld and integrate Path-Facing (PF) and Timestep-aware Gating (TAG) into it. Under identical training and inference settings at 640×320, adding PF and TAG improves all metrics over the reproduced EvoWorld baseline. After high-resolution fine-tuning at 1024×512, the enhanced model also outperforms the official EvoWorld.

Overall, these results demonstrate that our designs generalize across different backbones and can effectively benefit an independent panoramic framework. We will include these comparisons in the revised manuscript.

Table I. Comparison on EvoWorld's outdoor Unity_curve dataset. 25F and 73F denote the first-segment (25 frames) and three-segment (73 frames) evaluations, respectively. PF: Path-Facing Canonicalization; TAG: Timestep-Aware Gating. All reproduced models follow the same training schedule: 5K steps (batch size 24) at 640×320, followed by 3K fine-tuning steps at 1024×512. Other settings follow the official EvoWorld implementation.

MethodResolution25F-SSIM↑25F-PSNR↑25F-LPIPS↓25F-FID↓25F-FID_cube↓25F-TransErr↓25F-RotErr↓73F-SSIM↑73F-PSNR↑73F-LPIPS↓73F-FID↓73F-FID_cube↓73F-TransErr↓73F-RotErr↓
EvoWorld (official, native)1024×5760.774518.930.323814.6519.700.76810.100.748016.880.419225.5323.683.46743.70
EvoWorld (official, resized)1024×5120.772319.060.321614.5419.700.76810.100.744216.960.418125.3023.683.46743.70
PVWander (SVD-based)1024×5120.819922.000.199112.0119.520.0690.2620.789120.040.259713.0217.421.79518.15
EvoWorld* (reproduced)640×3200.732119.260.303420.6827.510.5758.2060.690516.870.420536.0934.883.54045.12
EvoWorld* + PF & TAG640×3200.774521.190.215614.1120.770.2390.3380.741719.380.271516.9718.811.83618.27
EvoWorld* + PF & TAG1024×5120.809121.130.226511.3817.890.2330.2740.783419.430.270511.5015.061.82718.32

W2: Gains of two designs

1. For Tab. 3

Compared with online memory alone, adding Gating or Path-Facing individually decreases SSIM by 0.0036 and 0.0039, respectively, while both consistently improve LPIPS (0.4132 → 0.4042 / 0.4107), FID (26.69 → 23.79 / 23.09), and FID_cube (42.67 → 40.06 / 40.50).

This trade-off may be because the generated details, although perceptually clearer, do not necessarily align spatially with the ground truth. SSIM measures local luminance, contrast, and structural similarity, making it sensitive to local correspondence; sharper or more natural-looking results therefore do not always yield higher SSIM.

To illustrate this discrepancy, we provide qualitative comparisons in Section 1 of Anonymous Project Page. As shown in Fig. 1 of page, the online-memory baseline exhibits substantial smoothing and detail degradation, yet achieves higher SSIM than the Gating output (0.6244 vs. 0.6010), despite worse LPIPS (0.6180 vs. 0.4577). Furthermore, as shown in Fig. 2, applying Gaussian blur to the same Gating output frame increases SSIM to 0.6332 while worsening LPIPS to 0.5781. Similar discrepancies between structural similarity metrics and perceptual quality have also been reported in prior studies [1, 2].

Therefore, generated video quality should be assessed by considering complementary metrics that capture different aspects of structural fidelity and perceptual quality.

Importantly, Path-Facing and Gating exhibit complementary effects when combined: the full model improves all metrics over online memory alone, including SSIM. This demonstrates that their combination achieves a better balance between structural similarity and perceptual quality.

[1] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR 2018

[2] Image Quality Assessment: Unifying Structure and Texture Similarity. TPAMI 2022

2. For analysis of Tab.5, please see the next comment due to page limit.

3. For memory and gating

Online 3D memory accumulates errors from generated observations and depth estimates. Timestep-aware Gating mitigates their impact by modulating reprojected features across denoising steps to retain structural guidance and suppress unreliable details. It is designed to mitigate, not eliminate, accumulated errors; severe geometric inaccuracies can still cause structural deviations, as illustrated in Fig. 7 of the Limitation Section in our manuscript. This is a common challenge faced by methods that rely on 3D memory reprojection for guidance.

Despite this limitation, when memory provides reasonably reliable geometry, reprojection offers valuable structural guidance beyond pose-only generation, which is also demonstrated by Tab. 3 in our manuscript. This highlights the potential of 3D memory-guided generation, while improving geometric reliability remains an important direction for future research.

4. Path-Facing on specifically designed trajectories

Handling different motion patterns. Path-Facing determines a shared orientation using PCA on current entire trajectory. When no reliable principal direction can be identified, it falls back to the first frame's orientation. For rotation-only motion, since the camera center remains fixed, target views can be directly obtained through spherical reprojection of the existing panorama, without requiring video generation.

Evaluation on specifically designed trajectories. To evaluate Path-Facing under specifically designed motion patterns, we use the 3DGS of 98 test split from InteriorGS-OmniRoam to render 240 trajectories: 80 U-turns, 80 closed-return trajectories with near-zero net displacement, and 80 zig-zag trajectories, each 81 frames.

To isolate Path-Facing, we compare three configurations: (A) the Wan2.1-Fun baseline; (B) Path-Facing applied only at inference time (zero-shot); and (C) Path-Facing applied during both training and inference.

Table V. Evaluation of Path-Facing under challenge trajectories.

ConfigurationSSIM ↑PSNR ↑LPIPS ↓FID ↓
A: Baseline0.772523.840.08769.58
B: + PF, zero-shot0.773623.880.08729.21
C: + PF, train & infer0.776023.950.08759.03

As shown in Tab. V, despite frequent changes in local motion directions and ambiguous dominant translation directions, Path-Facing maintains generation performance comparable to or better than the baseline across these trajectories.

While Fig. 1 of the original manuscript shows that Path-Facing aligns motion directions with ERP orientations across most of 31K sampled clips, Tab. V further demonstrates that it remains applicable even when the dominant-motion-direction assumption is challenged by input trajectories.

W2: Gains of two designs

2. For Tab. 5

Following Reviewer's advice, we conduct multi-seed experiments for the zero-shot Path-Facing transfer in the original Tab. 5, repeating the evaluation with 3 random seeds on 4 models and reporting the mean and standard deviation in Tab. II. Note that OmniRoam Preview&Refine is excluded from current experiment, as evaluating it once on 98 test scenes requires about 800 A100 GPU hours, making multi-seed evaluation infeasible within the rebuttal period.

Table II. Zero-shot transfer of Path-Facing across panoramic generators. Results are reported as mean ± standard deviation over three random seeds. Results before and after Path-Facing are separated by arrows.

MethodSSIM ↑PSNR ↑LPIPS ↓
Wan2.1-Fun0.5613 ± 0.0119 → 0.5726 ± 0.012415.30 ± 0.30 → 15.70 ± 0.180.4248 ± 0.0045 → 0.4210 ± 0.0056
PanoWorld0.4353 ± 0.0193 → 0.4458 ± 0.018911.49 ± 0.56 → 11.65 ± 0.620.6009 ± 0.0093 → 0.5968 ± 0.0076
OmniRoam (Preview)0.5793 ± 0.0077 → 0.5804 ± 0.007614.84 ± 0.37 → 14.89 ± 0.350.4416 ± 0.0059 → 0.4363 ± 0.0048
Matrix3D0.5503 ± 0.0287 → 0.5514 ± 0.036212.56 ± 0.36 → 12.60 ± 0.420.5900 ± 0.0144 → 0.5884 ± 0.0173

As shown in Tab. II, applying Path-Facing yields consistent directional improvements in the mean across all metrics and 4 panoramic generators.

Importantly, these gains are obtained by applying Path-Facing in a zero-shot manner, without additional training for models. The purpose of this experiment is to demonstrate the potential of Path-Facing to transfer across different panoramic generators without additional training. Such zero-shot improvements are not trivial to achieve. For example, as shown in Tab. III, directly injecting 3D memory render latents into the pretrained Wan2.1-Fun model severely degrades generation quality, and training adaptation is required to effectively utilize the additional geometric guidance. In contrast, as shown in Tab. IV, applying Path-Facing during both training and inference brings larger improvements to Wan2.1-Fun compared with zero-shot. This demonstrates that Path-Facing not only offers plug-and-play transferability but can also bring further benefits through training-time adaptation.

Table III. Effect of different 3D memory injection strategies on Wan2.1-Fun. Zero-shot directly adds the 3D memory render latent to the noisy video latent without additional training. Train & Infer follows the setting in Table 6 of Appendix A.6 of the original manuscript. Due to the severe degradation under zero-shot injection, we report results on the first 81-frame segment only.

Method81F-SSIM ↑81F-PSNR ↑81F-LPIPS ↓
Wan2.1-Fun0.718120.880.1717
Wan2.1-Fun + 3d memory, zero-shot0.311310.250.7374
Wan2.1-Fun + 3d memory, train&infer0.735721.410.1654

Table IV. Effect of different Path-Facing (PF) application strategies on Wan2.1-Fun. All configurations use the same random seed. Zero-shot applies PF only at inference without additional training, while Train & Infer is Wan2.1-Fun setting in Table 3 of the original manuscript.

MethodSSIM ↑PSNR ↑LPIPS ↓
Wan2.1-Fun0.549215.360.4220
Wan2.1-Fun + PF, zero-shot0.560015.640.4201
Wan2.1-Fun + PF, train&infer0.576115.910.4094

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.