acceptodds
Under review as a conference paper at ICLR 2027

Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion

Abstract

Video Diffusion Models (VDMs) synthesize high-quality videos, but precise camera and motion control remains challenging when large viewpoint shifts, changing occlusions, or complex dynamics reveal unseen regions. Diffusion priors alone can yield structural inconsistencies, while warped guidance can contain incomplete geometry and misleading view-dependent appearance cues. Real2SAM2Real complements VDM priors with a coarse 3D foreground cache of topologically complete meshes for intuitive editing and animation. Generative 3D lifting estimates these object and human meshes, including unobserved surfaces, from a single reference image. Normal maps rendered under specified camera trajectories and foreground motions guide video synthesis in the VDM, while the reference image provides appearance conditioning. Soft spatial-aligned injection and minimally invasive fine-tuning incorporate this guidance while largely preserving pretrained synthesis priors. Decoupled conditioning enables independent appearance and geometry editing. We train on around 2,000 monocular clips using instance-masked normal sequences with instance-level spatial perturbations to reduce sensitivity to differences between video-estimated training normals and normals rendered from coarse 3D caches at inference. Training requires neither explicit 3D assets nor paired multi-trajectory recordings. Real2SAM2Real supports decoupled camera and multi-instance motion control, including articulated humans. Across these challenging settings, experiments show strong spatiotemporal consistency and better overall control accuracy and video fidelity than evaluated state-of-the-art methods. Preliminary demonstrations suggest potential robotics applications.

Primary area:
generative models
Keywords:
Controllable Video Generation, 3D Geometric Guidance, Camera and Motion Control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ymGTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Real2SAM2Real renders foreground meshes lifted from a reference image into normal-map conditions for camera, human, and object motion control. It separates appearance from geometry and uses lightweight fine-tuning with normal perturbations, adapting the model on approximately 2,000 monocular clips.

Critical strengths

1)The paper combines editable, complete foreground geometry with separate appearance conditioning to provide a unified and practical framework for camera, human, and object control. This integration of explicit control with video generation is the main reason I lean toward acceptance.

2)Results across multiple tasks and component ablations provide useful empirical support. Table 2 reduces PA-MPJPE from 34.81 mm for the strongest listed baseline to 22.80 mm, and Table 3 shows substantial improvements in object control; updating only approximately 1.8% of parameters per expert further strengthens the practical value.

Critical weaknesses

1)Appendix C.1 uses output lengths ranging from 25 to 121 frames and computes RPE between consecutive frames, resulting in unmatched trajectory intervals across methods. This reduces confidence in the corresponding camera rankings and should be addressed through matched-interval evaluation and explicit GLOMAP registration-failure accounting.

2)The ablations examine injection and perturbations but do not isolate the benefit of complete geometry. Without a controlled comparison against reference-visible-only caches, the explanation of how geometry completion contributes remains incomplete, although the overall system has empirical support.

Questions for authors

1)Were camera trajectories resampled to common timestamps before computing RPE? If not, do the relative rankings hold when evaluated over matched trajectory intervals, and how are GLOMAP registration failures handled?

2)Did the Uni3C evaluation in Table 2 enable its human-animation branch and use SMPL-X motion conditions? Appendix C.2 explicitly lists only camera trajectories and scene point clouds.

2 comments

We sincerely thank the reviewer for the careful review and insightful feedback.

C1. RPE comparability across different sequence lengths

In the paper's evaluation, we indeed aligned all videos to the same frames number before computing the camera metrics by following WorldForge’s evaluation protocol (EVO library default setting). The writing description there is fixed by adding the adjective ‘frame-aligned’. sorry for the confusion. To further address your concern, we additionally evaluate all methods with 49-frame alignment, which samples the trajectories more densely. ViewCrafter is excluded because it generates only 25 frames. As shown in the table, our method still achieves good and reliable camera accuracy under this denser alignment.

GLOMAP successfully registered over 99.99% of all frames. Only 11 frames, all from WorldForge videos with severe visual inconsistency. We excluded these frames from metric computation, which has a negligible impact on the results.

MethodATE (↓)RPE-T (↓)RPE-R (↓)
ReCamMaster0.06010.01550.419
TrajectoryCrafter0.04460.01140.568
WorldForge0.12590.04791.012
InSpatio-World0.09740.03030.464
GEN3C0.02130.00890.256
Ours0.02400.00650.205

 

C2. The necessity of comparing the complete mesh with the initial visible surfaces

Restricting our cache to the visible surfaces would reduce our method to a reconstruction-and-warping design like WorldForge. Such designs struggle in diverse cases (Fig. 2, Table 1). Our method departs from them on purpose.Therefore, we view this comparison as contrasting two design philosophies rather than necessary ablation study. Our existing evaluation therefore contrasts coarse but instance-complete geometry with pixel-aligned reconstructions containing only visible surfaces, using each method’s native representation and training strategy.

Warping the initially visible normals of our coarse proxy would remove its geometric completeness without providing the pixel alignment of a reconstruction-based proxy. This would discard the main benefit of our representation while retaining its approximation errors. Testing this modified condition without retraining would also introduce a training–inference mismatch, since our model is trained with foreground normals from the current view at each time step.

 

C3. Uni3C human-animation configuration

Yes. The Uni3C results reported in Table 2 were obtained with its human-animation branch enabled and SMPL-X motion conditioning. Appendix C.2 currently lists camera trajectories and scene point clouds but omits the human-motion conditioning used in this evaluation. We have revised this section to explicitly document the enabled branch and SMPL-X inputs (Highlighted in yellow).

Thank you for the clarifications and the additional evaluation.

The camera-metric concern is largely addressed. The confirmation that videos were already frame-aligned, together with the denser 49-frame evaluation and the accounting for the 11 unregistered WorldForge frames, increases my confidence in the reported camera rankings. The Uni3C setup in Table 2 is also now clear: the human-animation branch was enabled with SMPL-X motion conditioning, and the appendix has been updated accordingly.

The discussion of complete versus visible-only geometry is only partly resolved. I understand that a visible-surface cache would change the method’s design and that testing this condition without retraining would introduce a training–inference mismatch. At the same time, this remains a conceptual argument rather than a controlled comparison that isolates the benefit of geometry completion, so that part of the original concern is not fully closed.

Overall, the main evaluation issues have been addressed, and I will keep my rating of 3 (Weak acceptance).

Reviewer_LNymOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Real2SAM2Real, a framework that uses an editable 3D foreground cache to control camera and object motion in video generation. Starting from a reference image, pretrained 3D models construct object and human meshes, including estimates of unobserved surfaces. Normal maps rendered under the specified camera trajectories and foreground motions provide geometric guidance to a Wan2.2 video model, while the reference image supplies appearance information. Background synthesis largely relies on the pretrained video prior. The framework is adapted through a zero-initialized geometry embedding branch and LoRA. Instance-level spatial perturbations help bridge the gap between estimated training normals and mesh-rendered inference normals. Experiments cover camera control, human motion reconstruction, and object manipulation, with promising control accuracy. However, the individual contributions of complete geometry, normal-map conditioning, and the injection mechanism, as well as the comparability of some results, require further validation.

Critical strengths
  1. Editable meshes provide an intuitive interface for 3D control and geometric guidance for newly revealed surfaces. Separating normals from reference appearance can help avoid misleading view-dependent cues introduced by directly reprojecting colors, particularly around reflections and refractions.

  2. The method constructs training pairs from approximately 2,000 monocular clips and trains approximately 1.8% of the parameters per expert. This offers a practical approach to combining existing 3D and video generation models.

  3. The evaluation includes 200 camera-control cases, 300 human-motion clips, and 100 object-manipulation cases. The reported PA-MPJPE of 22.80 mm in Table 2 outperforms the compared methods, while Table 3 shows substantial reductions in object rotation and translation errors.

  4. The perturbation ablations and cache-displacement experiments provide initial evidence of robustness to coarse geometry. The paper also acknowledges limitations involving unobserved appearance, identity drift under prolonged occlusion, and complex contact and deformation.

Critical weaknesses
  1. The distinction from closely related work and the benefits of the core representation are not sufficiently established. Shape-for-Motion already uses editable meshes, rendered normals, and separate geometry and appearance controls, including an image-to-video animation application. The paper does not adequately discuss this closely related work or provide matched-backbone comparisons of complete versus visible-only geometry and normals versus alternative geometric conditions.

  2. The interpretation of the Soft Spatial-Aligned Injection ablation is incomplete. As described, the module adds a linear projection of geometry to the input patch embeddings. For aligned linear projections, Wxx+WnnW_xx + W_nn can be expressed as a linear projection of channel-concatenated inputs. The initialization, freezing policy, and training configuration of the concatenation baseline therefore need clarification before the performance difference can be attributed to additive injection itself.

  3. Some tasks lack recent, directly relevant baselines. Camera control would benefit from comparison with InSpatio-World. Object control is compared only with DaS and ViewCrafter, although the latter is not designed specifically for this task; relevant alternatives include VerseCrafter and FlexTraj. Differences in backbones, geometric completeness, and conditioning representations also limit attribution of the gains to the proposed design.

Questions for authors
  1. Could the authors compare complete meshes with geometry restricted to surfaces visible in the reference view, and compare normal, depth, and trajectory conditioning under the same backbone, data, and training budget? What is the key methodological advance over Shape-for-Motion, and what evidence isolates its contribution?

  2. How is the concatenation baseline initialized, and which parameters are frozen or trained? Does a performance gap remain when the original input weights are preserved and only the additional geometry weights are trained under matched settings?

  3. Could the authors include InSpatio-World for camera control and VerseCrafter or FlexTraj for object control? For methods supporting mesh-sampled trajectories, could the same SAM 3D geometry be provided to distinguish the contribution of geometric input quality from that of the generation method?

8 comments

We sincerely thank the reviewer for the careful review and insightful feedback.

Concern 1: Key differences from Shape-for-Motion

Our novelty is to show that a coarse 3D cache, complete but imprecise, is sufficient to control video generation from a single image. Shape-for-Motion, as a video object editing pipeline, follows the opposite philosophy. It reconstructs an accurate 3D proxy from the input video and trains its model to strictly follow the geometry. We instead train the video model to use the cache as a coarse spatial anchor: the 3D specifies where each instance is and how it moves, while the 2D reference image supplies the exact shape, appearance at specific key moments. As detailed below, this balance makes coarse 3D sufficient for control while leaving generation unconstrained, so the output still follows the geometry yet keeps details beyond 3D proxy.

The newly added Fig. 9 on page 10 of the revised PDF provides a direct qualitative comparison with Shape-for-Motion using the same coarse body proxy. The proxy lacks the hat and loose hair and exhibits shoulder misalignment. At frames 16 and 60, Shape-for-Motion distorts or loses the hat brim and loose hair. Our method follows the body motion while better preserving these details beyond the proxy silhouette. It also synthesizes fine dynamics, such as hair moving in the wind, that are difficult to reconstruct accurately in 3D. With proxy geometry held fixed, this comparison illustrates our framework’s ability to use imperfect geometry for motion control while synthesizing appearance and dynamics beyond what the proxy explicitly represents.

The key differences lie in the intended role of the 3D proxy and the scope of control. Shape-for-Motion primarily targets object-centric video editing: it reconstructs a temporally consistent mesh of a selected object from an input video, propagates edits through this proxy, and uses its geometry and texture renderings to synthesize the edited video. Its pipeline therefore emphasizes faithful reconstruction of the object being edited. User-specified global camera trajectories are not an explicitly evaluated control setting.

Real2SAM2Real is designed for video creation from a single reference image, with joint control over camera trajectories and individual foreground motions. Its 3D cache provides coarse, editable spatial and motion guidance without needing to precisely reconstruct the reference appearance. The reference image supplies appearance, while the VDM synthesizes fine visual and dynamic details without requiring the proxy to explicitly encode textures or these details. These may include clothing and accessories, facial expressions, and secondary motion such as hair or fabric movement. This decoupling allows coarse geometry to drive detailed appearance and dynamics without requiring precise correspondence between the proxy surface and the generated visual content.

This is particularly valuable for video creation. Users can obtain a rough foreground mesh from a single image or animate a parametric body model, then compose camera and foreground motions without modeling every appearance detail and secondary deformation. This reduces the geometric modeling burden and makes motion creation and revision more accessible.

We thank the reviewer for highlighting Shape-for-Motion. We have included and cited it in the Related Work section, with a discussion clarifying the distinctions above.

Concern 2: Comparison with the additional methods —FlexTraj, and VerseCrafter, and InSpatio-World

We have added comparisons with FlexTraj and VerseCrafter in Figs. 10 and 11 of the revised PDF, respectively.

For object motion control, FlexTraj receives trajectories derived from the same SAM 3D meshes used by our method (Fig. 10), as suggested by the reviewer. This controls the source geometry while retaining each method’s native conditioning representation. In the illustrated examples, FlexTraj exhibits distortions in object shape and appearance during motion, whereas our results better preserve object structure and reference appearance. This robustness is consistent with our training design: the model is explicitly trained to synthesize detailed appearance under coarse geometric guidance, with normal-map perturbations encouraging tolerance to geometric inaccuracies.

MethodVBench Avg.RotErrTransErr
DaS81.0622.762548.1813
FlexTraj81.9713.175133.6757
Ours82.034.193212.0991

 

For dynamic human scenes, Fig. 11 provides a comparison with VerseCrafter. VerseCrafter represents each object with a single 3D Gaussian, providing control over overall position, extent, and ellipsoid orientation, but not explicitly encoding part-level articulation. For approximately axisymmetric human proxies, rotations can produce similar rendered controls, making the intended facing direction ambiguous. Our body normal maps explicitly represent articulated surface geometry, providing more detailed guidance for limb motion and body orientation.

MethodVBench Avg.PA-MPJPE (mm)AKD-2D (px)MKR (%)
Uni3C74.3659.45108.5817.85
VerseCrafter77.7458.0064.441.95
Ours77.3522.8017.800.58

 

For camera control, we additionally compare with InSpatio-World:

MethodVBench Avg.ATERPE-TRPE-R
ViewCrafter79.380.08090.07123.6352
WorldForge77.600.12810.06052.2211
InSpatio-World78.700.09720.03111.2423
Ours80.560.02340.01321.0940

We will include these additional results in the final version of the paper. We thank the reviewer for suggesting these relevant baselines and helping us strengthen the evaluation.

Concern 3: The necessity of comparing the complete mesh with the initial visible surfaces

Restricting our cache to the visible surfaces would reduce our method to a reconstruction-and-warping design like WorldForge. Such designs struggle in diverse cases (Fig. 2, Table 1). Our method departs from them on purpose.Therefore, we view this comparison as contrasting two design philosophies rather than necessary ablation study. Our existing evaluation therefore contrasts coarse but instance-complete geometry with pixel-aligned reconstructions containing only visible surfaces, using each method’s native representation and training strategy.

Warping the initially visible normals of our coarse proxy would remove its geometric completeness without providing the pixel alignment of a reconstruction-based proxy. This would discard the main benefit of our representation while retaining its approximation errors. Testing this modified condition without retraining would also introduce a training–inference mismatch, since our model is trained with foreground normals from the current view at each time step.

Furthermore, the added FlexTraj comparison in Fig. 10 uses the same SAM 3D meshes as our method, providing a comparison with a common source of geometry.

 

Concern 4: The necessity of comparing across different conditioning modalities, like normals, depth, and trajectories

We respectfully consider the requested ablations to be complementary analyses, rather than necessary experiments for supporting the main contribution of this paper. Our contribution is a framework that uses coarse, editable, instance-complete foreground geometry for controllable video creation. Normal maps serve as the interface for conveying this geometry to the VDM; we do not claim that normals are universally superior to depth or trajectory conditioning.

Regarding conditioning modalities, comparing normals, depth, and trajectories would investigate alternative implementations of geometric guidance. Trajectory conditioning also requires choices about point identities, sampling density, temporal correspondence, and visibility handling, extending beyond a change of rendering modality. Such a study would be valuable for understanding alternative designs, but establishing a ranking among these modalities is not a claim or objective of this paper. We therefore believe the existing evaluations, together with the additional baseline comparisons, provide appropriate evidence for the contribution being claimed. We will clarify this scope in the paper and distinguish the underlying 3D representation from its conditioning interface. We thank the reviewer for encouraging a clearer articulation of these distinctions.

 

Concern 5:linear projection of channel-concatenated vs. Soft Spatial-Aligned Injection

The differences reported in Fig. 7 and Tables 1 and 3 remain when the pretrained input weights are preserved and frozen in both variants. Both use zero-initialized geometry weights and train only these additional weights together with the same LoRA parameters. The backbone, reference-image conditioning pathway, training data, and optimization settings are otherwise identical.

Under the matched budget of approximately 2,000 video clips and 6,000 training steps, Fig. 7 shows that the concatenation variant produces green and purple patterns on the car resembling the normal-map visualization, whereas our full model better preserves the reference colors and surface details. The quantitative ablations in Tables 1 and 3 further support improvements in control accuracy and video quality for camera and object control, respectively.

We agree that aligned additive projections and channel concatenation followed by a linear projection are mathematically equivalent. Our evidence therefore concerns the observed behavior of the tested implementations under this training budget, rather than greater expressive capacity of addition.

Our design uses coarse geometry as spatial and motion guidance, without requiring precise alignment with the reference appearance. The aim is to preserve reference appearance, including details absent from the proxy. These results support our choice of Soft Spatial-Aligned Injection for the training regime studied here. They do not establish an intrinsic optimization advantage of addition, and whether the observed difference persists with substantially more data or training remains untested. We have clarified the baseline configuration and the scope of the ablation’s conclusions in the revised PDF (Highlighted in yellow). We thank the reviewer for prompting this clarification.

Reviewer_69gTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Real2SAM2Real, which extends video diffusion models with an editable 3D foreground cache created from only a single reference image. Geometric guidance comes from normal maps that are rendered from the image, whereas the appearance is provided by the reference image. This is achieved through soft spatial-aligned injection and minimal fine-tuning on about 2,000 monocular clips using spatial perturbations for dealing with imperfect geometry. The design would support camera control, manipulation of objects, and even animation of humans while leaving geometry and appearance editing independent.

Critical strengths
  1. Reported motion-control gains over the reported papers is substantial.
  2. The ablations serve as a valid proof of the effectiveness of normal map perturbations in dealing with sub-optimal 3D guidance.
  3. The paper provides a good contribution to methodology by integrating an editable 3D cache which facilitates the intuitive editing of both camera and object movement independently of their appearance.
Critical weaknesses
  1. Limited comparison to newer methods like MTVCraft.
  2. The number of frames generated ranges from 25 to 121, while RPE is calculated between successive frames. Such intervals could correspond to varying degrees of camera motion. The researchers need to test RPE along the same trajectory interval to confirm comparability of the rankings.
  3. While Appendix D perturbs objects, it doesn’t evaluate incorrect shapes or missing geometry. Measurements by PSNR, SSIM, and LPIPS from Appendix D are performed only on the first frame in the reference-conditioned manner. Although camera mistakes and VBench offer some extra evidence, the robustness to viewpoint changes and disocclusion is less direct.
Questions for authors
  1. The number of frames generated ranges from 25 to 121, while RPE is calculated between successive frames. Such intervals could correspond to varying degrees of camera motion. Please test RPE along the same trajectory interval to confirm comparability of the rankings.
  2. Could you compare complete meshes versus initially visible surfaces using the same model and conditioning pipeline, in order to test the effect of geometry completion under large viewpoint changes and occlusions? This would help isolating the role played by geometry completion in the improved results of control accuracy and visual coherence.
1 comment

We sincerely thank the reviewer for the careful review and insightful feedback.

C1. Comparison with MTVCrafter

We have added a qualitative comparison with MTVCrafter in Fig. 11 of the revised PDF. In the illustrated cases involving large motion, MTVCrafter exhibits noticeable degradation in visual quality, whereas our method produces results with better visual fidelity. We will include these results in the final version. We thank the reviewer for suggesting this relevant baseline.

 

C2. RPE comparability across different sequence lengths

In the paper's evaluation, we indeed aligned all videos to the same frames number before computing the camera metrics by following WorldForge’s evaluation protocol (EVO library default setting). The writing description there is fixed by adding the adjective ‘frame-aligned’. sorry for the confusion. To further address your concern, we additionally evaluate all methods with 49-frame alignment, which samples the trajectories more densely. ViewCrafter is excluded because it generates only 25 frames. As shown in the table, our method still achieves good and reliable camera accuracy under this denser alignment.

MethodATE (↓)RPE-T (↓)RPE-R (↓)
ReCamMaster0.06010.01550.419
TrajectoryCrafter0.04460.01140.568
WorldForge0.12590.04791.012
InSpatio-World0.09740.03030.464
GEN3C0.02130.00890.256
Ours0.02400.00650.205

 

C3. Geometric mismatch, the scope of Appendix D

We would like to clarify that geometric mismatch is already part of our standard evaluation setting. Our evaluations use coarse proxies without requiring precise shape correspondence with the reference appearance. Fig. 9 illustrates this setting: the human proxy does not represent features such as the hat and loose hair, yet our generated results retain these details. Thus, tolerance to imperfect geometry is evaluated under our normal operating conditions, beyond the artificial perturbations in Appendix D.

Appendix D serves a different purpose: it is a stress test that deliberately increases geometric misalignment to examine its effect on generation quality. It is not intended as a targeted evaluation of robustness to viewpoint changes or disocclusion. Its reference-frame PSNR, SSIM, and LPIPS measure fidelity at that view, rather than the quality of newly revealed regions.

 

C4. The necessity of comparing the complete mesh with the initial visible surfaces

Restricting our cache to the visible surfaces would reduce our method to a reconstruction-and-warping design like WorldForge. Such designs struggle in diverse cases (Fig. 2, Table 1). Our method departs from them on purpose.Therefore, we view this comparison as contrasting two design philosophies rather than necessary ablation study. Our existing evaluation therefore contrasts coarse but instance-complete geometry with pixel-aligned reconstructions containing only visible surfaces, using each method’s native representation and training strategy.

Warping the initially visible normals of our coarse proxy would remove its geometric completeness without providing the pixel alignment of a reconstruction-based proxy. This would discard the main benefit of our representation while retaining its approximation errors. Testing this modified condition without retraining would also introduce a training–inference mismatch, since our model is trained with foreground normals from the current view at each time step.

Comments on the paper

1 comment

We thank all reviewers for their valuable feedback and the Area Chairs for their time and effort in overseeing the review process.

Our updates are organized as follows:

  1. Manuscript revisions. We have updated the paper in response to the reviewers’ comments. All revisions are highlighted with a yellow background to make them easy to locate.

  2. Additional visual comparisons. All newly added visual comparisons are collected in Figs. 9–11 on page 10 of the revised PDF for easy reference. These figures are also referenced in our responses to the relevant reviews.

  3. Additional quantitative results. These results are presented in tables within the corresponding reviewer responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.