acceptodds
Under review as a conference paper at ICLR 2027

Preserve, Reveal, Expand: Towards Faithful 4D Video Editing with Region-Aware Conditioning and Benchmarking

Abstract

While 4D-driven video diffusion models can generate visually plausible videos, faithful 4D editing poses a stricter requirement: source-observed content should be preserved, while disoccluded and out-of-view regions must be synthesized naturally. We identify a key failure mode in existing methods, which we term Evidence-Role Mismatch: source-backed observations, uncertain rendered cues, and unsupported regions are entangled within a single conditioning signal, which can lead to preservation drift, ghosting, and unstable extrapolation. To mitigate this issue, we propose PREX (Preserve, Reveal, Expand), a region-aware framework that decomposes the target spatiotemporal volume into three distinct roles based on observation support and scene extent. PREX traces edited 4D points back to source frames and retrieves observed colors and pairs these appearance cues with geometry-aware confidence and region roles, and injects them into a frozen video diffusion backbone through an adapter trained with self-supervised proxy editing tasks. We further introduce PREBench, a diagnostic benchmark specifically designed for 4D video editing, to expose and quantify such region-specific failures. PREBench separately evaluates Preserve fidelity, Reveal quality, and Expand plausibility of videos generated by 4D driven video diffusion models. Experiments show that PREX substantially reduces region-specific failure modes while maintaining strong visual quality and 4D edit control compared with existing 4D-based video diffusion methods. Code will be publicly released.

Primary area:
generative models
Keywords:
4D Video Editing, Region-Aware Conditioning, Video Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zx9hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses 4D video editing, where a monocular video is lifted to a 4D point cloud, edited (camera trajectory and/or object transforms), and re-rendered as a coarse condition for a video diffusion model. The authors propose PREX, which partitions every target pixel into Preserve, Reveal and Expand. PREX retrieves appearance directly from source frames, computes a continuous confidence map, and injects these together with Reveal/Expand masks into a frozen model through a trainable adapter. The authors also introduce PREBench, a benchmark with nine region-specific diagnostic metrics.

Critical strengths
  1. The paper breaks 4D editing into sub-problems of preserve, reveal and extend. Building on this decomposition, the authors construct PREBench, covering camera-only and joint camera/object edits, with failure-specific metrics for each region type.

  2. Practical training strategy that requires no paired editing data. The self-supervised proxy tasks sidestep the requirement of paired editing dataset by artificially removing 4D support from source videos.

Critical weaknesses
  1. Heuristic P/R/E partition.

(1) Every target pixel must be exactly one of Preserve, Reveal, or Expand. This significantly limits the application to general 4D editing which should be able to edit both camera and motion. Current method is only a small extension to novel view video generation by enabling to transform object’s position and size. For example, they exclude non-rigid object deformation, articulated motion changes, and appearance or illumination changes.

(2) Most synthesis is reduced to background completion. It is unclear how PREX behaves when the text prompt asks for new content in the expanded region rather than a plain continuation of the background. No metric measures prompt adherence in Expand, and E-Copy only penalizes copying boundary texture.

(3) The Reveal/Expand boundary is confusing and inconsistently defined. In Eq. 15 the authors define Reveal and Expand with scene geometry In lines 877–880 the authors instead reassigns unsupported connected components by assigning a component touching an 8-px image-border band to Expand, otherwise to Reveal. When does each rule hold? And when do they disagree? The paper does not state which rule produces the final masks used for conditioning and for PREBench, nor how often the two disagree.

  1. Validity of PREBench.

(1) Proxy errors become ground truth. Masks and references are derived from the edited 4D proxy, not a ground truth edited target. If the proxy contains errors, a method that faithfully copies these artifacts scores well, while one that corrects them with its generative prior is penalized.

(2) R-Ghost only rewards dissimilarity to the ghost reference, and E-Temp only rewards low temporal change. Blurry or near-static completions can still score well. No metric measures whether the Reveal/Expand content is plausible or semantically correct.

Questions for authors
  1. The Related Work groups ReCamMaster with methods that "render or generate videos from new viewpoints by warping source observations and inpainting disoccluded regions" (lines152–154). As far as I understand, ReCamMaster does not warp: it conditions the generator on the source video and camera parameters directly, without an explicit 3D proxy.

  2. The human validation of PREBench is too small to support the claim that the metrics reflect human judgment. Each of the nine metrics is validated on only 30 pairs rated by 3 annotators, and the aggregate score on 30 pairs rated by 5 annotators. At this scale, the reported agreements carry wide uncertainty.

  3. PREX doesn’t outperform baselines on the camera-only split. On the full set, the remaining comparators are either not designed for source-preserving editing (Yume, Perception-as-Control) or use different and weaker backbones. This raises the question of whether PREX is particularly strong on object edits, or whether object edits are simply harder to evaluate fairly. In joint camera/object cases, the Reveal regions, dynamic-preserve masks, and ghost references all depend on the edited proxy, the same information PREX receives as input. Could the authors also report Table 2 separately for the joint camera/object split for all full-set methods?

Reviewer_Er9VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PREX, which is a method to edit camera and obj motion in videos and preserve the original scene's appearance. It reconstructs a 4D scene and splits output pixels into 3 bins:

  • Preserve where there is source observation which is valid.
  • Reveal for where the missing parts must be completed withing the captured scene.
  • Expand is where content must be generated beyond that scene. Source appearance, confidence estimates, and region masks are used to guide a trainable adapter connected to the frozen diffusion video model. They use self-supervised proxy tasks from ordinary videos to avoid the need for paired editing examples. The authors also propose PREBench which is a benchmark with multiple editing cases and separate measures for preservation, completion, and expansion. Their validations cover baseline comparisons and component ablations and human preference analysis. They generally achieve better performance than the baselines on the proposed benchmark.
Critical strengths
  • the matched baseline is convincing evidence for the performance of the method. The comparison uses the exact same backbone and first frame conditioning and training data and etc and also standardizes the budget. PREX still reduces P-LPIPS and improves VBench.
  • Preservation remains stronger when observations are not fed (appendix). This actually is to address a concern I had at first, which is if preservation scorse otherwise would benefit (in an unfair way) from directly using the images used as reference.
  • Source appearance retrieval directly addresses the editing problem. It is systematically determined that which source observation is useful/trustable. This is also ablated.
  • The distinction between reveal and expand is directly tested. Merging them into one unsupported region actually increases the R-Ghost and E-Seam.
  • Having to collect expensive paired editing data is smartly avoided by the training strategy. The proxy tasks change the evidence available and keep the original video as supervision. This is also well-ablated.
  • It is great that the results are supported by human preferences. The size of the participants pool is also meaningful and valid.
Critical weaknesses
  • R-Ghost doesn't necessarily or specifically measure if ghost content is removed. The score improve swhen the output is less similar to the invalid rendered reference. So changing the color of a remaining ghost could improve the score without fixing the artifact.
  • R-Seam can possibly miss local boundary issues. It compares the mean RGB colors of two boundary bands, and discards spacial arrangement. Some strong discontinuities (local) can cancel when averaged while an actual color diff across a boundary can increase the score.
  • Aggregate-score calibration partly uses test outputs. The ref scales used to computed PREBench score are fitted using examples from both the test split and calibratoin set. It is unclear if there is still overlap in scale fitting in the separate overall-pref study.
Questions for authors

My questions would be basically to address the weaknesses above.

  • Clarification of metric validity.
  • Clarification of calibration independence.
  • Clarification of supported edits. Should the edit preserve the original cam and object configuration in the first frame? If not, how is an edited initial confi used with the original first-frame condition, and how is it evaluated?
Reviewer_BFSSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses 4D video editing, including novel-view synthesis through camera trajectory changes and object editing through motion and placement changes. PREX partitions target-view regions into Preserve, Reveal, and Expand, and conditions a video diffusion model on these masks, source-backed appearance cues, and confidence maps. The paper also introduces PREBench for region-specific evaluation. Results show improvements in overall VBench scores and many proposed diagnostic metrics, with qualitative examples demonstrating reduced ghosting.

Critical strengths
  • Compared with previous methods that use rendering masks for conditioning, the paper further distinguishes Preserve, Reveal, and Expand regions. This is well motivated and makes the diffusion model more aware of the different generation goals.
  • The evaluation results looks promising. PREX achieves better overall VBench scores and improves many region-specific metrics. The provided visual examples also show fewer ghost artifacts around edited objects.
Critical weaknesses
  • Some PREBench metrics may not faithfully reflect generation quality. Preserve metrics compare generated content with covisible source observations without accounting for view-dependent appearance differences between the source and target views. PSNR and SSIM could complement the perceptual metrics, although they share this limitation. In Reveal regions, R-Ghost decreases whenever the output becomes less similar to the invalid rendering reference, even if the completion is implausible. R-Seam compares mean boundary-band colors, so spatially inconsistent boundaries can receive low scores when their means match.
  • Although the paper describes the supported editing operations, it does not sufficiently specify how the benchmark’s object edits were selected, their magnitudes, or the number of cases per edit type, in both train and test set. These details would help assess the difficulty of the joint camera/object editing task and the range of edits evaluated.
Questions for authors

Please see the concerns raised in the critical weaknesses section above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We sincerely thank all reviewers for their detailed reviews and constructive feedback. We have made the following changes to the paper and uploaded a new revision (all updates are highlighted in blue in the revised PDF):

  • Joint Camera/Object split evaluation (Table 2, Section 5.1): Added separate quantitative comparisons on the 200-case Joint Camera/Object subset across all full-set baselines and the matched Plain Wan2.1-ControlNet (Reviewer zx9h).
  • Pixel-level Preserve metrics (Section 4, Appendix B.1.1, Appendix D.2 Table 11): Added P-PSNR and P-SSIM results on both the Camera-only and Full sets alongside discussions on view-dependent appearance and view-alignment optimization (Reviewer BFSS).
  • Test-set-exclusive score calibration & human validation CIs (Tables 2 & 4, Appendix B.1.4, Appendix C Tables 5–7): Updated all main-paper PREBench Scores to use anchors calibrated strictly on the independent calibration set (hmcalh_m^{\text{cal}}, 0% test-set outputs), moved the original combined-anchor scores to Table 6 alongside calibration-free mean ranks (Reviewer Er9V), and added 95% Wilson confidence intervals and gap-stratification analysis in Table 7 (Reviewer zx9h).
  • Dataset edit statistics & unified mask pipeline (Section 5, Appendices A.3, A.4 & H): Added case counts and rotation/translation/scaling magnitude statistics for the 150 Camera-Only and 200 Joint test cases in Appendix H.2, linked the 10,000 training pairs to Appendix A.4 (Reviewer BFSS), and unified the Reveal/Expand mask construction steps into Appendix A.3 (Reviewer zx9h).
  • Diagnostic metric scopes & evaluation decoupling (Section 4, Section 5.1, Appendices B.1.2, B.1.3 & D): Clarified the diagnostic roles and mutual safeguards of R-Ghost, R-Seam (vs. E-Seam), and E-Temp (Reviewers BFSS, Er9V, zx9h), summarized the four safeguards against 4D proxy errors, and discussed the interplay between PREX's conditioning inputs and the evaluation protocol (Reviewer zx9h).
  • Related work & editing scope clarifications (Section 2, Section 6, Appendices A.1 & G): Clarified the positioning of ReCamMaster in Section 2 (Reviewer zx9h), the modularity of upstream 4D proxies for non-rigid edits in Section 6 and Appendix A.1 (Reviewer zx9h), and the roles of source-caption text prompts and first-frame I2V conditioning in Appendix G (Reviewers Er9V, zx9h).

Overall, we are pleased that the reviewers found our method sound and well-presented. Thank you once again! Your feedback has been invaluable in improving our paper, and we believe these revisions will strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.