acceptodds
Under review as a conference paper at ICLR 2027

TTA4D: Object-Centric Free-Viewpoint Video Generation via One-Shot Test-Time Adaptation

Abstract

Generating free-viewpoint videos of a dynamic object from a single monocular video requires temporal coherence and view-consistent geometry, yet explicit 4D methods often distort geometry on in-the-wild inputs, and controllable video models drift from the input's appearance under large viewpoint changes. We present TTA4D, a one-shot test-time adaptation framework that converts a pre-trained image-to-video diffusion model into an object-specific free-viewpoint video generator in approximately 10–15 minutes on a single GPU. Our key observation is that the missing supervision, a moving object seen from a moving camera, factorizes into two readily available signals: a dynamic-object fixed-camera signal from the input video, which teaches the model to follow geometric conditioning as the object moves, and a static-object dynamic-camera signal rendered from a single-image 3D reconstruction, which teaches it to follow conditioning as the viewpoint changes. After LoRA adaptation on both, the model composes them along user-specified object-centric trajectories while preserving the input's appearance and motion. Because TTA4D adds only LoRA weights, without architectural changes or paired control data, it is directly applicable to newly released video foundation models. With this adaptation, a plain image-to-video model outperforms the depth-controllable VACE given the same geometry, and adapting VACE itself yields further improvement. On Consistent4D, TTA4D with the VACE backbone outperforms six baselines on all four metrics for both seen and unseen trajectories, and its outputs can directly supervise a 4D Gaussian Splatting reconstruction. On 22 real-world DAVIS videos, even its weakest configuration, a plain image-to-video model with coarse voxel geometry, receives a mean rating of in a blinded user study, compared with for the strongest baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generative Model, Novel View Synthesis, 4D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PAfPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TTA4D is a one-shot test-time adaptation framework that uses LoRA fine-tuning (~15 mins) to turn image-to-video diffusion models into free-viewpoint video generators. It factorizes supervision into dynamic-object/fixed-camera and static-object/dynamic-camera signals, enabling arbitrary camera trajectory control while preserving object appearance and motion to outperform strong baselines on Consistent4D and DAVIS datasets.

Critical strengths
  • The proposed one-shot test-time adaptation strategy that factorizes supervision into dynamic-object and dynamic-camera signals to achieve novel viewpoint control without architectural changes or external paired data.
  • Because the adaptation relies solely on lightweight LoRA fine-tuning, the method is lightweight (10–15 minutes on a single GPU) and directly plug-and-playable across various image-to-video foundation backbones.
  • The method shows strong empirical performance over a number of sota baselines.
Critical weaknesses
  • While the evaluation is conducted on 22 DAVIS videos, could the authors clarify whether this scale is standard and sufficient relative to existing literature benchmarks? We look forward to seeing how this addresses feedback from other reviewers.
  • Given a fixed monocular input video, does the method require LoRA fine-tuning for each distinct user-specified camera trajectory? If fine-tuning is not performed per trajectory, the mechanism enabling generalization to unseen trajectories requires further explanation. An ablation study examining the impact of NN (Figure 2) on generalization capability would help clarify this.
  • Lines 105 to 109 discussing accurate geometry require revision, as the current phrasing lacks sufficient context and is difficult to follow.
  • To allow reviewers and readers to properly assess reconstruction and rendering quality, we recommend providing interactive or video visualizations of the 4D Gaussians.
  • How does the pipeline handle failure cases or inaccuracies in the output generated by the TRELLIS model? Could the authors explicitly discuss how sensitivity to foundational generation errors affects overall stability as a limitation of the framework?
  • Conditioning the TRELLIS-based rendering exclusively on the first frame introduces potential failure modes—such as initial frame artifacts or foreground object occlusion. Furthermore, while later frames in the video may provide richer multi-view appearance cues, these could potentially conflict with the initial TRELLIS output. How do the authors address or mitigate potential inconsistencies between the single-frame reconstruction and subsequent video evidence?
  • Another potential concern is that the proposed method relies heavily on pre-trained multi-view generative priors, raising questions regarding the fairness of the baseline comparisons.
Questions for authors
  • I believe L85-86 (It's highly likely that VFM's training data include videos featuring moving object from moving camera) is not true. Rather they were not conditioned with geometric conditions. If authors agree, this statement has to be fixed.
  • The paper's writing assumes readers are already aware of details of SS4D and VACE. There should be at least one line opener sentence describing what each is.
5 comments

We thank the reviewer for the constructive feedback and for recognizing the factorized supervision and the effectiveness of test-time adaptation. We address each point below.

[W1] Scale and scope of the real-world evaluation

Our evaluation on 22 real-world DAVIS videos is consistent with established practice and adequate in scale relative to comparable object-centric 4D generation studies. The table below summarizes the real-world evaluation datasets used in representative related work, placing our evaluation scale in context.

MethodVenue / yearReal-world evaluation
L4GMNeurIPS 20242 real ActivityNet videos(Appendix G.1)
SV4D 2.0ICCV 202520 DAVIS videos (§5.3)
GVFDiffusionICCV 2025None, Only Synthetic
SS4DACM TOG 202514 DAVIS videos (§5.2)
Lift4DSIGGRAPH Asia 20268 DAVIS + 10 Pexels videos (§4.2)
TTA4DThis submission22 DAVIS videos; (§4.1)

[W2] Trajectory generalization

No per-trajectory fine-tuning is required. TTA4D learns one set of LoRA weights per input video and reuses it across camera trajectories (Sec. 3.3, [L 256]). TTA4D’s dual-signal supervision guides the pretrained video model to follow changes in conditioning caused by object and camera motion. Generalization to unseen trajectories comes from transferring this learned behavior to new conditioning sequences, rather than simply memorizing a fixed set of camera paths.

Number of SODC trajectories NNUnseen PSNR ↑Unseen SSIM ↑Unseen LPIPS ↓Unseen IoU ↑
218.7040.8980.1350.718
3 (Default)19.6570.9070.1120.734
419.6630.9070.1110.729

We conducted the ablation on the number of SODC trajectories using the I2V backbone. Increasing the number of SODC trajectories improves performance on held-out trajectories, but our default set of three already performs well in most cases, with limited gains from adding further trajectories.

[W3] Clarifying lines 105–109

We thank the reviewer for highlighting this clarity issue. We will revise the passage around lines 105–109:

“Geometric conditioning specifies the target structure, but preserving the input object’s appearance also requires effective use of the reference image. To examine whether object-specific adaptation improves appearance fidelity beyond existing depth control, we compare VACE (Jiang et al., 2025), a video model supporting depth and reference-image conditioning, before and after TTA4D adaptation. With identical World Tracing geometry and the same reference image, adaptation reduces LPIPS from 0.129 to 0.091 (Table 3), demonstrating improved appearance fidelity under the same geometric conditioning.”

[W4] 4D Gaussian visualizations

We agree that video is more informative than static frames for assessing motion and rendering quality. Figure 4 currently shows RGB renderings and Gaussian primitives of the fitted DAVIS camel, and the supplementary demo includes a short dynamic visualization. Due to the page limit, we will host additional interactive or video visualizations of the 4D Gaussians on the project page.

[W5, W6] TRELLIS errors and consistency with later video frames

We thank the reviewer and address W5 and W6 together. TRELLIS renderings provide SODC supervision for viewpoint changes; they are not used as the reference image at inference. Instead, inference uses the real first frame, while later input frames provide appearance and motion supervision through DOFC during adaptation.

Since SODC also provides RGB supervision, TRELLIS errors can still influence the adapted model. Nevertheless, our qualitative results and visual comparisons generally show consistent appearance and motion with the input video, supporting the effectiveness of this factorized supervision in practice. We will add comparisons with later input frames, including failure cases, and explicitly discuss the remaining sensitivity to TRELLIS errors as a limitation.

[W7] Pretrained priors and fairness of comparison

Pretrained 3D and video priors individually leave gaps in object-centric 4D generation; TTA4D’s contribution is to coordinate these complementary priors through dual-signal adaptation. Using pretrained priors is also common among the related methods, as summarized below.

Method3D / Video prior
L4GMLGM
SS4DTRELLIS
GVFDiffusionTRELLIS
Lift4DSAM3D
TrajectoryCrafterCogVideo
GEN3CCosmos

[Q1] What supervision is missing in lines 85–86?

We will make two edits at L85–86:

  • Replace “supervision” with “instance-specific supervision.”
  • Replace “the target configuration, a moving object viewed from a moving camera, is never observed” with “paired target videos of the test object undergoing its recorded motion from the requested novel viewpoints are unavailable.”

[Q2] Introducing SS4D and VACE

  • At L98–99, we will add: “SS4D is a generative model that produces dynamic 3D objects from monocular video; we use its coarse voxel sequences to obtain inference-time depth and mask conditioning.”
  • At L105–107, before the VACE comparison, we will add: “VACE is a pretrained video generation and editing framework supporting reference-image and depth-based control; we use it as an optional controllable video backbone.”
Reviewer_PAfPOpenReview ↗

Response from Reviewer

Thanks the authors for the rebuttal. The rebuttal partially addresses my conern while other points are kept as future work or promises. For example, why not the manuscript is updated with colored texts on the fly? Moreover, the author addresss W6 by future planned experiments. I would like to see other reviewer's perspectives and discuss with the reviewer for this submission.

Reviewer_jmsFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles the problem of 4D novel view synthesis with test-time adaptation in an object-centric setting. The idea is that one can teach an image-to-video diffusion model novel-view synthesis by decoupling the task into a static camera moving object and moving camera static object synthesis tasks.

Critical strengths
  • the problem of 4D novel-view synthesis is important in general
  • the test-time training setup is interesting in general
Critical weaknesses
  • there are many SOTA baselines missing like GCD, CogNVS, CAT4D (has a very similar premise as this work), DressRecon including many standard benchmarks like DNA-Rendering, ActorsHQ, or even Objaverse, DeformableObjects4D etc. The paper is not thorough in their empirical analysis.
  • There are many design decisions in the paper that are not motivated properly — for example why is depth conditioning important, what is the role of adding object masks when there is only a single object in the video with a white background? What does a mask tell the model more?
  • If the paper is about novel-view synthesis, how would the masks and depth maps be supplied as conditioning input for the novel views at inference?
  • There are two models being trained which are both bulky and are diffusion based which means that they would need minutes to infer novel-views from, but there is no runtime comparison in the paper against all the other baseline methods considered.
  • The entire pipeline is really heavy — you need to run DAv3, SAM2, TRELLIS and then two DiTs and then 4DGS. I am not able to justify this design for the marginal gain in novel-view synthesis performance.
  • What happens when you run 4DGS on top of other diffusion model outputs too — like TrajectoryCrafter? How does this compare with per-frame 3DGS and other standard optimization based 4DGS methods like ShapeofMotion etc. many others.
  • How do the two DiTs agree with each other when synthesizing the appearance of regions that are not seen in input? What in the training or inference enforces this?
  • How are the camera poses and timestamps fed into the model?
Questions for authors

see above

5 comments

We thank the reviewer and address each point below. New results: CogNVS and GCD baselines, 5 additional Objaverse objects, 4DGS on TrajectoryCrafter outputs, per-frame 3DGS, and an end-to-end runtime table.

[W1] Missing baselines

"there are many SOTA baselines missing like GCD, CogNVS, CAT4D (has a very similar premise as this work), DressRecon"

The paper already compares with six baselines, including recent state-of-the-art methods for object-centric 4D generation such as Lift4D (SIGGRAPH Asia 2026). CogNVS (NeurIPS 2025) and GCD (ECCV 2024), mentioned by the reviewer, predate this newer baseline; nevertheless, we have also compared with both methods.

For CAT4D, we could not identify publicly released model code and weights for a reproducible quantitative comparison. DressRecon (3DV 2025) targets clothed-human reconstruction and incorporates human-specific shape and pose priors. It has released training and rendering code, together with preprocessed human sequences, but the official release does not include preprocessing scripts for custom videos. Applying its released pipeline to our mixed-category dynamic-object benchmark would require implementing the missing preprocessing and adapting the default human-specific initialization.

The added results below show that TTA4D outperforms CogNVS and GCD on all reported metrics for both seen and unseen trajectories.

MethodSeen PSNR↑SSIM↑LPIPS↓IoU↑Unseen PSNR↑SSIM↑LPIPS↓IoU↑
CogNVS18.7290.9010.1330.67618.6730.8940.1370.658
GCD17.9400.8920.2000.44917.5290.8860.2190.449
TTA4D20.8420.9260.0840.74720.4820.9160.0980.740

[W2] Missing benchmarks

"including many standard benchmarks like DNA-Rendering, ActorsHQ, or even Objaverse, DeformableObjects4D etc."

  • DNA-Rendering, ActorsHQ. Human capture datasets, outside our general-object setting. General-object 4D methods do not use them (L4GM: Consistent4D; GVFDiffusion: Consistent4D and Objaverse-XL), nor do GCD, CogNVS, or CAT4D.
  • Objaverse. Consistent4D's objects are animated Objaverse assets, so Table 1 already covers Objaverse. We add 5 more animated Objaverse objects, with the same protocol as Table 1:
MethodSeen PSNR↑SSIM↑LPIPS↓IoU↑Unseen PSNR↑SSIM↑LPIPS↓IoU↑
GVFDiffusion18.8340.9140.1030.52218.2110.9030.1180.505
L4GM19.8800.9170.1140.69019.6250.9030.1070.609
SS4D19.0160.8950.1090.58219.1570.9400.1120.607
GEN3C18.7240.8800.1720.65617.9130.8610.1970.634
TTA4D20.4040.9250.0860.74920.0510.9140.1040.735
  • DeformableObjects4D. We could not find this dataset and would be glad to evaluate on it given a reference.

[W3] Thoroughness

"The paper is not thorough in their empirical analysis."

The paper reports six baselines on eight trajectories with four metrics (Table 1, L270–286, L366–373), ablations of both signals, two geometry sources, and two backbones (Table 3, L400–417, L444–450), a blinded user study with 20 participants on 22 real videos (Table 2, L388–396, L892–901), and downstream 4DGS (Sec. 4.5, L452–463). The rebuttal adds the results above.

[W4] Why depth conditioning

"why is depth conditioning important"

Depth is the interface that lets the two signals compose. (i) Object motion (DOFC) and camera motion (SODC) change the rendered depth in the same form, so the model can follow both jointly at inference (Sec. 3.2, L248–254). (ii) A plain I2V model has no camera input; depth enters through its built-in image conditioning, so only LoRA weights are added (Sec. 3.3, L263–268, L289–293), whereas pose conditioning requires a camera-controlled model trained on paired data. (iii) Depth can be rendered from any geometry at any viewpoint.

Empirically, this depth-conditioned adaptation enables a plain I2V backbone to outperform zero-shot VACE with the same geometry (Table 3, L444–450), while the full TTA4D configuration outperforms all six evaluated baselines on all four metrics for both seen and unseen trajectories (Table 1, L270–286).

[W5] Role of masks

"what is the role of adding object masks when there is only a single object in the video with a white background? What does a mask tell the model more?"

Our real-world evaluation (DAVIS) has cluttered backgrounds, where masks isolate the object. Even with a uniform background (Consistent4D): (i) pixels without geometry have no depth, so without a mask "no surface" is ambiguous with "distant surface"; the mask gives the silhouette explicitly. (ii) Our depth normalization uses foreground quantiles (Sec. 3.1, L224). For I2V, masks use the native conditioning mask M_cond (Sec. 3.3, L263–268) at no extra cost.

[W6] Conditioning at inference

"how would the masks and depth maps be supplied as conditioning input for the novel views at inference?"

As specified in Sec. 3.1, Eq. (2) (L222), and Fig. 2(b) (L187), depth and masks are rendered from dynamic geometry (SS4D voxels or World Tracing) along the target trajectory, (Dtπ,Mtπ)=Render(Ot,π)(D_t^π, M_t^π) = Render(O_t, π), with the first input frame as reference.

[W7] Number of trained models

"There are two models being trained which are both bulky and are diffusion based"

TTA4D trains a single model: "Only LoRA weights of the frozen video diffusion transformer (DiT) are trained" (Fig. 2 caption, L180); "One adapted model serves any number of trajectories" (Sec. 3.3, L256). The two DiT boxes in Fig. 2 show the same model at adaptation and inference.

[W8] Runtime

"there is no runtime comparison in the paper against all the other baseline methods considered"

Adaptation takes ~10–15 minutes per object (Sec. 4.1). Full breakdown (one RTX 6000 Ada GPU, 24 frames):

StageTimePeak memory
Depth (DA3) + masks (SAM 2)45 s12 GiB
Single-image 3D (TRELLIS.2) + SODC rendering3 min20 GiB
Inference geometry (SS4D voxels / WT)4 min25 GiB
LoRA adaptation (once per video)15 min43 GiB
Sampling (per trajectory)3 min34 GiB
BaselineTimePeak memory
GVFDiffusion8 min13 GiB
L4GM5 min13 GiB
SS4D12 min11 GiB
Lift4D90 min20 GiB
TrajectoryCrafter23 min27 GiB
GEN3C30 min43 GiB
CogNVS720min (1 GPU)30 GiB
GCD3 min15 GiB

† The official CogNVS implementation reports approximately 70 minutes for 200 fine-tuning steps on 8 A6000 Ada GPUs. This is author-reported adaptation time, excluding preprocessing and sampling, and uses a different hardware configuration from our single-GPU measurements.

Faster pretrained baselines, such as L4GM and GCD, learn their task offline and require no per-video model updates. TTA4D also uses pretrained models but performs an additional, one-time object-specific LoRA adaptation. Lift4D and CogNVS likewise incur per-video optimization or adaptation, making them closer comparisons for instance-specific fitting cost. Offline pretraining costs are excluded for all methods.

[W9] Pipeline cost and gains

"The entire pipeline is really heavy — you need to run DAv3, SAM2, TRELLIS and then two DiTs and then 4DGS. I am not able to justify this design for the marginal gain in novel-view synthesis performance."

  • We do not consider these gains marginal. PSNR +1.83 / +0.95 dB over the strongest baseline (seen / unseen) and +3.15 dB over TrajectoryCrafter; LPIPS 0.084 vs. 0.105 (Table 1). User study: 4.56 vs. 3.26, with our weakest configuration (Table 2).
  • Off-the-shelf components are standard. TrajectoryCrafter uses DepthCrafter; GEN3C uses Depth Anything V2 plus SLAM poses for video input; Lift4D chains SAM 3, SAM 3D, and a diffusion prior in a test-time optimization.

[W10] 4DGS on other outputs, per-frame 3DGS, Shape of Motion

"What happens when you run 4DGS on top of other diffusion model outputs too — like TrajectoryCrafter? How does this compare with per-frame 3DGS and other standard optimization based 4DGS methods like ShapeofMotion etc."

  • The 4DGS pipeline is not our main contribution. We use a standard 4DGS only to test the consistency of the generated videos (Sec. 4.5).
  • TrajectoryCrafter + 4DGS (added below). The quality of generated novel-view videos substantially affects downstream 4DGS reconstruction quality.
  • Per-frame 3DGS (added below). Without temporal correspondence, it flickers and breaks motion coherence.
  • Shape of Motion reconstructs only observed surfaces and cannot complete unseen regions (e.g., the back) needed for 360° orbits (Sec. 2).
MethodSeen PSNR↑LPIPS↓Unseen PSNR↑LPIPS↓
TrajectoryCrafter + 4DGS16.6580.16216.3260.171
TTA4D + per-frame 3DGS17.4500.15917.2700.163
TTA4D + 4DGS20.9010.08420.5710.096

[W11] Agreement between "two DiTs"

"How do the two DiTs agree with each other when synthesizing the appearance of regions that are not seen in input? What in the training or inference enforces this?"

There is a single model (see [W7]). Unseen regions stay consistent because all outputs share their sources: one 3D model supervises these regions through SODC, all trajectories use geometry from one point-cloud sequence OtO_t, and all share the reference frame I1I_1. The 4DGS results (Sec. 4.5, [W10]) confirm cross-view consistency.

[W12] Camera poses and timestamps

"How are the camera poses and timestamps fed into the model?"

Neither is fed explicitly; this is why we condition on depth.

  • Pose enters through the depth and masks rendered at πtπ_t (Eq. 2, L222).
  • Timestamps. The model needs whether and how the object moves, which depth encodes, not absolute time. A frozen instant and a motionless object over time give identical conditioning and should give identical output; SODC (Eq. 4, L240) is exactly this case, and fixing OtO_t while varying πtπ_t yields bullet-time video. (The diffusion timestep uses the backbone's native embedding.)

We will incorporate these results and clarifications into the revision.

Reviewer_UDGDOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper proposes TTA4D, a one-shot test-time adaptation framework for object-centric free-viewpoint video generation from a single monocular video. It combines dynamic-object fixed-camera (DOFC) supervision from the input video with static-object dynamic-camera (SODC) supervision from single-image 3D reconstruction. Through LoRA adaptation, TTA4D enables pretrained image-to-video models to generate novel-view videos using depth and mask conditioning. Experiments on Consistent4D and DAVIS demonstrate improved generation quality and object consistency over existing methods.

Critical strengths

1.The decomposition into DOFC and SODC provides a practical way to construct complementary training signals without requiring ground-truth videos of moving objects under moving cameras.
2.TTA4D enables a standard image-to-video model to follow geometric conditioning through LoRA adaptation, without modifying its architecture or requiring a separately trained controllable model.
3.The method outperforms baselines on Consistent4D across PSNR, SSIM, LPIPS, and IoU for both seen and unseen camera trajectories.
4.The qualitative comparisons and blinded user study on DAVIS suggest improved appearance preservation and object consistency under large viewpoint changes.

Critical weaknesses
  1. Although Table 3 demonstrates the benefit of combining DOFC and SODC supervision, it lacks a SODC-only baseline and does not fully isolate their individual and interaction effects. More direct evidence is needed to establish that the model learns to compose object motion and camera motion rather than simply benefiting from additional viewpoint supervision.
  2. Dependence on single-image 3D priors. SODC supervision relies on a 3D model reconstructed from the first frame, whose unseen geometry and appearance may be inaccurate. The paper lacks a systematic analysis of how reconstruction errors affect the final videos, particularly for complex objects, unusual poses, and occlusions.
    3.The quantitative evaluation mainly relies on frame-level reconstruction metrics. Although the user study and downstream 4DGS results are encouraging, more direct measurements of temporal coherence and cross-view geometric consistency are needed to support the main claims.
  3. The DAVIS evaluation focuses on videos with a dominant foreground object and limited occlusion, while experiments are restricted to 24-frame clips. The method's robustness to severe occlusions, large non-rigid deformations, topology changes, and longer videos remains unclear.
Questions for authors

1.Table 3 demonstrates the benefit of combining DOFC and SODC, but lacks a SODC-only baseline. A more complete ablation would help clarify their individual contributions and interaction effects.
2.The method relies on single-image 3D reconstruction, but its robustness to inaccurate geometry remains unclear. How do errors in occluded or unseen regions affect the final generation quality?
3.The evaluation mainly relies on frame-level reconstruction metrics and downstream 4DGS results. More direct quantitative measurements of temporal coherence and cross-view geometric consistency would strengthen the claims.
4.Could the authors evaluate TTA4D on more challenging videos involving severe occlusions, large non-rigid deformations, and longer temporal sequences?
5.Could the authors report the end-to-end computational cost, including geometric preprocessing, LoRA adaptation, video generation, and peak GPU memory?
6.How well does TTA4D handle severe occlusions, large non-rigid deformations, and longer video sequences? Additional experiments would help establish its robustness in these settings.
7.The paper reports the LoRA adaptation time but does not fully characterize the end-to-end computational cost. It would be helpful to report the runtime of geometric preprocessing and video generation, as well as peak GPU memory usage. 8. The fitted 4DGS should be evaluated on azimuths excluded from its supervision set, with separate results for fixed-view and spin180 trajectories. This would help determine whether the experiment demonstrates genuine cross-view consistency beyond fitting the generated views. 9. A comparison with ReCapture under a consistent object-centric protocol and mutually supported camera trajectories would strengthen the evaluation. If such a comparison is infeasible, the specific limitations should be clarified.

7 comments

We thank the reviewer for recognizing the complementary supervision signals and the effectiveness of adapting a standard image-to-video model. We address the requested ablations, robustness analysis, consistency evaluation, computational cost, and ReCapture comparison below.

[W1, Q1] Isolating DOFC and SODC

We thank the reviewer for this constructive point. We report the requested compute-matched SODC-only ablation on both backbones below. SODC-only already includes novel-view supervision, yet adding DOFC improves PSNR by 1.867/0.912 dB and reduces LPIPS from 0.142/0.108 to 0.109/0.091 for I2V/VACE, respectively. Conversely, adding SODC to DOFC improves PSNR by 4.478/1.077 dB. Joint supervision achieves the best results across all four metrics on both backbones. These comparisons show that viewpoint supervision alone does not account for the full improvement and support the complementary contributions of the two signals to generating dynamic objects under changing viewpoints.

BackboneAdaptation signalsPSNR↑SSIM↑LPIPS↓IoU↑
I2VNone15.1470.8470.3800.642
I2VDOFC only15.2890.8520.1950.673
I2VSODC only17.9000.8960.1420.708
I2VDOFC + SODC19.7670.9110.1090.739
VACENone19.1680.9030.1290.723
VACEDOFC only19.5850.9090.1110.730
VACESODC only19.7500.9160.1080.733
VACEDOFC + SODC20.6620.9210.0910.744

[W2, Q2] Single-image reconstruction errors

Our experiments indicate that the adapted model does not simply reproduce errors in the first-frame 3D reconstruction. Instead, it primarily learns to respond to changes in geometric conditioning driven by object and camera motion, while drawing appearance information mainly from the reference image provided at inference time. Although severe reconstruction errors can degrade the output, imperfections in the initial reconstruction generally have a limited impact on the quality of the generated videos.

[W3, Q3] Temporal coherence and cross-view geometric consistency

We have added a MEt3R-based evaluation using the same seen and unseen trajectories as in Table 1. We assess consistency across time and viewpoints using two frame-pairing protocols:

  • Temporal: MEt3R between consecutive frames within each generated video
  • Cross-view: MEt3R between frames generated at the same time step from different viewpoints
MethodSeen Temporal↓Seen Cross-view↓Unseen Temporal↓Unseen Cross-view↓
SS4D0.19110.15220.18590.2141
GVFDiffusion0.22470.18360.22330.2439
Lift4D0.19200.15940.18710.2117
L4GM0.18670.15520.18180.1987
TrajectoryCrafter0.28360.33170.30040.3118
GEN3C0.30610.36340.34470.3916
TTA4D0.17640.13670.17830.1755

TTA4D achieves the lowest MEt3R scores in all four settings. Relative to the best-performing baseline in each setting, it reduces cross-view MEt3R by 10.2% and 11.7% on seen and unseen trajectories, respectively, and temporal MEt3R by 5.5% and 1.9%. These results provide direct quantitative support for improved geometric consistency across adjacent time steps and viewpoints, complementing the existing user study and downstream 4DGS results.

[W4, Q4, Q6] Occlusions, large non-rigid deformation, topology changes and longer videos

Occlusion. We distinguish self-occlusion from external occlusion. Our focus is primarily on self-occlusion: generating stable and consistent views of object surfaces that are not visible in the input video. Severe, prolonged occlusion by other objects makes the problem closer to outpainting/completion rather than object-centric 4D generation. As stated in Sec. 4.1 [L 322], our DAVIS evaluation focuses on videos with a dominant foreground object and limited occlusion.

Non-rigid deformation and topology changes. Our paper already demonstrates articulated motion and large non-rigid deformation, such as dance-twirl, horsejump-high, and breakdance (Sec. F [L 903]). More complex dynamics involving topology changes, such as fluid splitting and merging, are beyond our current scope: the visible observations and static first-frame prior do not reliably constrain how the unseen geometry evolves.

Longer videos. Long-video generation depends largely on the video backbone. TTA4D adapts the model to follow dynamic depth conditioning without changing its temporal architecture. We use a common 24-frame evaluation window for consistent comparison. Related methods use comparable processing windows: L4GM uses 16-frame windows, while SS4D uses 32-frame. We will include longer-sequence results in the revision to illustrate performance beyond the evaluated 24-frame clips.

[Q5, Q7] End-to-end runtime and peak GPU memory

Adaptation takes ~10–15 minutes per object (Sec. 4.1). Full breakdown (one RTX 6000 Ada GPU, 24 frames):

StageTimePeak memory
Depth (DA3) + masks (SAM 2)45 s12 GiB
Single-image 3D (TRELLIS.2) + SODC rendering3 min20 GiB
Inference geometry (SS4D voxels / WT)4 min25 GiB
LoRA adaptation (once per video)15 min43 GiB
Sampling (per trajectory)3 min34 GiB

Adaptation is performed once per video; additional trajectories require only conditioning rendering and sampling.

[Q8] 4DGS evaluation outside its supervision views

The “unseen” split in Table 1 is defined with respect to SODC adaptation, rather than 4DGS fitting. We have now evaluated the fitted 4DGS on azimuths excluded from both initialization and fitting, with separate results for fixed-view and spin180 trajectories:

Held-out trajectoryPSNR↑SSIM↑LPIPS↓IoU↑
Fixed-view20.1620.9130.1020.735
Spin18019.5710.9080.1030.724

[Q9] Comparison with ReCapture

We agree that a comparison with ReCapture would be valuable. However, its official code has not been released, so we are unable to evaluate it under the same object-centric protocol.

Thank you very much for providing these new results. However, I am not entirely clear on the claim in your paper regarding the use of outputs from an adapted model to supervise the 4DGS reconstruction. While applying DOFC and SODC individually leads to significant performance gains when using an I2V (Image-to-Video) backbone, the improvements observed with the VICE backbone are limited; could you please explain why this is the case? Furthermore, given that the paper states DOFC and SODC capture object motion and camera motion respectively, how do you account for the coupling between them during the adaptation process?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.